2022-03-10T01:43:02.388 INFO:root:teuthology version: 1.1.0-ff5db916 2022-03-10T01:43:02.390 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728541 --owner scheduled_yuriw@teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} --verbose --name yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi -- /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728541/orig.config.yaml 2022-03-10T01:43:02.426 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-10T01:43:02.506 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728541 branch: wip-yuri5-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} email: ceph-qa@ceph.io first_in_suite: false job_id: '6728541' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-03-07-0958 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri5-testing-2022-03-07-0958 sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri5-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 targets: smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiq+m3MxFzr62uGf98OnW/IzZjWZNCWSiIyXhkJ9cCtkz3Bdp3TE8q9PSff3P7SiFZL61VzhCyHQ38Hx8zUW5AwI/6ismjGDldxM7XGTDS8BHBAWXj2IY9yUNwF+xAqlyD5okhurlWIdcbSswISdr/F10YfcoFxoSPfokl1/K4DORR2hziVcyas2zLt5zSFeji6IwOoDV/IyNxspUK33YZb2SWmEYqmFN3QKuPBtOB21OBSkKJMJslQTB6zEy8Qb0qO2i+JQslG/jvE0lleot05Ri/rS05GKTJO0xwMds7d+C21MxvI64ahUQJTw0u4qSmhIjqucPnT1/9o4M3cYL83X3D/1m2FY1NWBm4X9G2dgRlC1G72GI2dCIh6k2Z9gFjOHHfakufIzacr7g5Dxfui4o41UWvqs0xa1n+GDJJ0pfsvMI07IVOpq8gVTrpRWwFjAJstO3UUnUr2iqRPX0cp64ZaBSMM++HnYQAHyxY5kgMGKbT93YjO6EL4hbueVs= smithi071.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtct65vKv9I0xI/swkjZwKuLKyuS2IriqplG5A+oKStjsAisctHQXDqHwC0r3TZ8sGu7L9VtZwyUi8nqWQvjm2aP8cCnvC9JVOKflNio2JjJLzpDskucb0R8qwdqYgIlllFn6QPoTpCdTtPBtDHg1DUKkoNqXVQTNnafNEzjnZcenPYqLJw/+8kp6Umn8bLy/TD+V0GchscLDWV8rimh2q9uthOnXX6kpyxmIYQ35Xm3ZlbG1auGmIkPZ+2vCQSQZSI3a850nJbRLAf6KrQlzoqOh7Y1Wk20cwWkmGIU4OzjjHsvBSlhiaJ0wGUWc0hCzXfwEiG9Vc9J39K5eY7zMo/NK8Fa1/vkLZk8ngi0JhFMng7e44SIcTiVEiMDf2+TWWq7r0T2xJXafkb9KVxAGZ1SS9xSm8bxwAaJYEQ8vGHkHZs1+G7MQ7sATit6G13ZOh010Abf9xP3CEJlkd8PaSDq4cd9MGaWD9iVtcGfdeHyCj7bz3fa9hJbGZgeX6H5M= smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaH9AK/HjcrP+XK0gNW2zWirigKaE+RWQOqTbX6wULj7A89RLBZW7bnxZ8dD+h7woUPO84FiwvA8jKGbZYaQ5bv/DQdw4cvYcFTjh8+VDqgvXdXoSUXcCcQSsp+FL0o7g8QCD2+pnrYYfJfU++XXDfJJK1/Xg9ACVINI4IJvCL9J45D8Vl/93yHI1XSOmoFWmuJQPXGEuodO5IYhZkuS0sxcCvWzPkwu8PGrmDo2RIl8FGoh5VPkzawouKeCPHsXNDI3lkQUu2wNtT7AC37Y9PQNHmQZIq1lpebifPNYAer+1/KSIV316DhRkhiAvtEMCEccB32upkjd73W1bsHici5UAMx/cODKrbHaJ4tJEm28De86jZx8uVI6Zo8iajwnJDXttIzbGXQ8UfG2OGiLJCNkFXrkTJo/W0Bh6S2Ci9ygBiAhpuFYhPqP/IdGw6MYCkiykdm/3rWGah+iR7U5n+57g+Y436y79oc62COkZ1zXtiROl0s1Z6lSBP8RtBTSk= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: ff5db9169566776b2f41faf671316ca58a318ea6 timestamp: 2022-03-10_01:04:51 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-10T01:43:02.507 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa; will attempt to use it 2022-03-10T01:43:02.508 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks 2022-03-10T01:43:02.508 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-10T01:43:02.510 INFO:teuthology.task.internal:Checking packages... 2022-03-10T01:43:02.538 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7a44e9ebca556862e9704be295a07719d7db47b3' 2022-03-10T01:43:02.539 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-10T01:43:02.539 INFO:teuthology.packaging:ref: None 2022-03-10T01:43:02.540 INFO:teuthology.packaging:tag: None 2022-03-10T01:43:02.540 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-03-07-0958 2022-03-10T01:43:02.540 INFO:teuthology.packaging:sha1: 7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-10T01:43:02.541 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-03-07-0958 2022-03-10T01:43:02.718 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11025-g7a44e9eb-1focal 2022-03-10T01:43:02.718 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-10T01:43:02.727 INFO:teuthology.task.internal:no buildpackages task found 2022-03-10T01:43:02.728 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-10T01:43:02.748 INFO:teuthology.task.internal:Saving configuration 2022-03-10T01:43:02.764 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-10T01:43:02.783 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-10T01:43:02.808 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 01:37:26.702360', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE4BUkui4h0QWZDLDppJxSX3+8JfXFUkW3mQ4x3zjTH4'} 2022-03-10T01:43:02.832 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 01:37:26.704782', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP78AKlQ6l9rN1Dc0CBVdNvyfI+IX4E22i5HRWY+bt1F'} 2022-03-10T01:43:02.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-10_01:04:51-rados-wip-yuri5-testing-2022-03-07-0958-distro-default-smithi/6728541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-10 01:37:26.703719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMdC3Zm1iLHmrSTyWis46/Aybh2uolV3JCXyzsQlOTy'} 2022-03-10T01:43:02.854 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-10T01:43:02.865 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-10T01:43:02.865 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-10T01:43:02.866 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-10T01:43:02.866 INFO:teuthology.run_tasks:Running task console_log... 2022-03-10T01:43:02.978 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f22f7cc09d8>, signals=[15]) 2022-03-10T01:43:02.980 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-10T01:43:02.988 INFO:teuthology.task.internal:Opening connections... 2022-03-10T01:43:02.989 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2022-03-10T01:43:02.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:43:03.056 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2022-03-10T01:43:03.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:43:03.111 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2022-03-10T01:43:03.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:43:03.162 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-10T01:43:03.173 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2022-03-10T01:43:03.181 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2022-03-10T01:43:03.181 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2022-03-10T01:43:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2022-03-10T01:43:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T01:43:03.230 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2022-03-10T01:43:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2022-03-10T01:43:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T01:43:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="20.04" 2022-03-10T01:43:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T01:43:03.231 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T01:43:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T01:43:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T01:43:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=focal 2022-03-10T01:43:03.232 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=focal 2022-03-10T01:43:03.233 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2022-03-10T01:43:03.252 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2022-03-10T01:43:03.259 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2022-03-10T01:43:03.260 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2022-03-10T01:43:03.308 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="Ubuntu" 2022-03-10T01:43:03.309 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T01:43:03.309 INFO:teuthology.orchestra.run.smithi071.stdout:ID=ubuntu 2022-03-10T01:43:03.309 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE=debian 2022-03-10T01:43:03.310 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T01:43:03.310 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="20.04" 2022-03-10T01:43:03.310 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T01:43:03.310 INFO:teuthology.orchestra.run.smithi071.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T01:43:03.311 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T01:43:03.311 INFO:teuthology.orchestra.run.smithi071.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T01:43:03.311 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_CODENAME=focal 2022-03-10T01:43:03.311 INFO:teuthology.orchestra.run.smithi071.stdout:UBUNTU_CODENAME=focal 2022-03-10T01:43:03.312 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2022-03-10T01:43:03.331 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2022-03-10T01:43:03.339 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2022-03-10T01:43:03.339 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2022-03-10T01:43:03.387 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu" 2022-03-10T01:43:03.388 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-10T01:43:03.388 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu 2022-03-10T01:43:03.388 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian 2022-03-10T01:43:03.389 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-10T01:43:03.389 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="20.04" 2022-03-10T01:43:03.389 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-10T01:43:03.389 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-10T01:43:03.390 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-10T01:43:03.390 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-10T01:43:03.390 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=focal 2022-03-10T01:43:03.390 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=focal 2022-03-10T01:43:03.391 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2022-03-10T01:43:03.409 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-10T01:43:03.424 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-10T01:43:03.440 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-10T01:43:03.441 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2022-03-10T01:43:03.444 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2022-03-10T01:43:03.446 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2022-03-10T01:43:03.452 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-10T01:43:03.466 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-10T01:43:03.467 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2022-03-10T01:43:03.494 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2022-03-10T01:43:03.496 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2022-03-10T01:43:03.515 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-10T01:43:03.595 INFO:teuthology.run_tasks:Running task kernel... 2022-03-10T01:43:03.614 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-10T01:43:03.615 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-10T01:43:03.615 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-10T01:43:03.617 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-10T01:43:03.617 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2022-03-10T01:43:03.618 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2022-03-10T01:43:03.619 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2022-03-10T01:43:03.625 INFO:teuthology.orchestra.run.smithi071.stdout:5.4.0-97-generic 2022-03-10T01:43:03.626 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.4.0-97-generic 2022-03-10T01:43:03.626 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get clean 2022-03-10T01:43:03.628 INFO:teuthology.orchestra.run.smithi184.stdout:5.4.0-97-generic 2022-03-10T01:43:03.628 INFO:teuthology.orchestra.run.smithi007.stdout:5.4.0-97-generic 2022-03-10T01:43:03.629 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.4.0-97-generic 2022-03-10T01:43:03.629 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get clean 2022-03-10T01:43:03.630 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.4.0-97-generic 2022-03-10T01:43:03.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2022-03-10T01:43:03.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get update 2022-03-10T01:43:03.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2022-03-10T01:43:03.775 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get update 2022-03-10T01:43:03.966 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:43:03.967 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T01:43:04.028 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T01:43:04.078 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:43:04.078 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T01:43:04.086 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:43:04.087 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-10T01:43:04.123 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T01:43:04.141 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-10T01:43:04.366 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T01:43:04.443 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-10T01:43:04.470 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T01:43:04.495 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-10T01:43:04.543 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-10T01:43:04.555 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-10T01:43:04.586 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-10T01:43:04.686 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-10T01:43:04.694 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-10T01:43:04.830 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,310 kB] 2022-03-10T01:43:04.912 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-10T01:43:04.928 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-10T01:43:04.934 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-10T01:43:04.934 INFO:teuthology.orchestra.run.smithi071.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T01:43:04.993 INFO:teuthology.orchestra.run.smithi071.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-10T01:43:05.006 INFO:teuthology.orchestra.run.smithi071.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T01:43:05.008 INFO:teuthology.orchestra.run.smithi071.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T01:43:05.008 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [395 kB] 2022-03-10T01:43:05.010 INFO:teuthology.orchestra.run.smithi071.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T01:43:05.011 INFO:teuthology.orchestra.run.smithi071.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-10T01:43:05.023 INFO:teuthology.orchestra.run.smithi071.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T01:43:05.026 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-10T01:43:05.031 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,640 kB] 2022-03-10T01:43:05.032 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-10T01:43:05.033 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T01:43:05.034 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T01:43:05.035 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [228 kB] 2022-03-10T01:43:05.037 INFO:teuthology.orchestra.run.smithi071.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T01:43:05.038 INFO:teuthology.orchestra.run.smithi071.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T01:43:05.038 INFO:teuthology.orchestra.run.smithi071.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-10T01:43:05.039 INFO:teuthology.orchestra.run.smithi071.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T01:43:05.054 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-10T01:43:05.055 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-10T01:43:05.059 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [799 kB] 2022-03-10T01:43:05.064 INFO:teuthology.orchestra.run.smithi071.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-10T01:43:05.068 INFO:teuthology.orchestra.run.smithi071.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-10T01:43:05.104 INFO:teuthology.orchestra.run.smithi071.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T01:43:05.110 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T01:43:05.111 INFO:teuthology.orchestra.run.smithi071.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T01:43:05.111 INFO:teuthology.orchestra.run.smithi071.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-10T01:43:05.116 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T01:43:05.117 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-10T01:43:05.124 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [114 kB] 2022-03-10T01:43:05.130 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-10T01:43:05.130 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-10T01:43:05.133 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [615 kB] 2022-03-10T01:43:05.142 INFO:teuthology.orchestra.run.smithi071.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-10T01:43:05.142 INFO:teuthology.orchestra.run.smithi071.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T01:43:05.143 INFO:teuthology.orchestra.run.smithi071.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T01:43:05.143 INFO:teuthology.orchestra.run.smithi071.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T01:43:05.144 INFO:teuthology.orchestra.run.smithi071.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T01:43:05.150 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [681 kB] 2022-03-10T01:43:05.164 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T01:43:05.165 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-10T01:43:05.180 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T01:43:05.181 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T01:43:05.181 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-10T01:43:05.182 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T01:43:05.271 INFO:teuthology.orchestra.run.smithi071.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T01:43:05.272 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-10T01:43:05.275 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-10T01:43:05.277 INFO:teuthology.orchestra.run.smithi071.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-10T01:43:05.278 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-10T01:43:05.278 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T01:43:05.279 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T01:43:05.279 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T01:43:05.280 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T01:43:05.284 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T01:43:05.287 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T01:43:05.287 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-10T01:43:05.287 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [312 kB] 2022-03-10T01:43:05.289 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-10T01:43:05.289 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-10T01:43:05.290 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [853 kB] 2022-03-10T01:43:05.297 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-10T01:43:05.311 INFO:teuthology.orchestra.run.smithi071.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T01:43:05.312 INFO:teuthology.orchestra.run.smithi071.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T01:43:05.313 INFO:teuthology.orchestra.run.smithi071.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T01:43:05.313 INFO:teuthology.orchestra.run.smithi071.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T01:43:05.314 INFO:teuthology.orchestra.run.smithi071.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T01:43:05.315 INFO:teuthology.orchestra.run.smithi071.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T01:43:05.328 INFO:teuthology.orchestra.run.smithi071.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T01:43:05.328 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-10T01:43:05.329 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-10T01:43:05.330 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-10T01:43:05.330 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T01:43:05.331 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-10T01:43:05.331 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [122 kB] 2022-03-10T01:43:05.334 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-10T01:43:05.334 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-10T01:43:05.335 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-10T01:43:05.340 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T01:43:05.341 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T01:43:05.341 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T01:43:05.380 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [672 kB] 2022-03-10T01:43:05.406 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [202 kB] 2022-03-10T01:43:05.415 INFO:teuthology.orchestra.run.smithi071.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T01:43:05.415 INFO:teuthology.orchestra.run.smithi071.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T01:43:05.416 INFO:teuthology.orchestra.run.smithi071.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T01:43:05.417 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-10T01:43:05.417 INFO:teuthology.orchestra.run.smithi071.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T01:43:05.420 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T01:43:05.420 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T01:43:05.420 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T01:43:05.420 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T01:43:05.421 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T01:43:05.421 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,460 B] 2022-03-10T01:43:05.421 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-10T01:43:05.421 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-10T01:43:05.422 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-10T01:43:05.422 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-10T01:43:05.423 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T01:43:05.423 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T01:43:05.453 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-10T01:43:05.504 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T01:43:05.536 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-10T01:43:05.536 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-10T01:43:05.536 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-10T01:43:05.537 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-10T01:43:05.538 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-10T01:43:07.932 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 10.2 MB in 2s (4,712 kB/s) 2022-03-10T01:43:08.014 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 10.2 MB in 2s (4,709 kB/s) 2022-03-10T01:43:08.290 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 10.2 MB in 2s (4,393 kB/s) 2022-03-10T01:43:09.140 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2022-03-10T01:43:09.165 DEBUG:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T01:43:09.238 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2022-03-10T01:43:09.245 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2022-03-10T01:43:09.270 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T01:43:09.336 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2022-03-10T01:43:09.411 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2022-03-10T01:43:09.412 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2022-03-10T01:43:09.510 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2022-03-10T01:43:09.511 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2022-03-10T01:43:09.515 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-10T01:43:09.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T01:43:09.558 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2022-03-10T01:43:09.559 INFO:teuthology.orchestra.run.smithi071.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T01:43:09.559 INFO:teuthology.orchestra.run.smithi071.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T01:43:09.559 INFO:teuthology.orchestra.run.smithi071.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T01:43:09.560 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2022-03-10T01:43:09.560 INFO:teuthology.orchestra.run.smithi071.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T01:43:09.560 INFO:teuthology.orchestra.run.smithi071.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T01:43:09.604 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2022-03-10T01:43:09.605 INFO:teuthology.orchestra.run.smithi071.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T01:43:09.605 INFO:teuthology.orchestra.run.smithi071.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T01:43:09.605 INFO:teuthology.orchestra.run.smithi071.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T01:43:09.606 INFO:teuthology.orchestra.run.smithi071.stdout: thermald upower usbmuxd 2022-03-10T01:43:09.613 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-10T01:43:09.659 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2022-03-10T01:43:09.660 INFO:teuthology.orchestra.run.smithi184.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T01:43:09.660 INFO:teuthology.orchestra.run.smithi184.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T01:43:09.660 INFO:teuthology.orchestra.run.smithi184.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T01:43:09.661 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2022-03-10T01:43:09.661 INFO:teuthology.orchestra.run.smithi184.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T01:43:09.661 INFO:teuthology.orchestra.run.smithi184.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T01:43:09.711 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2022-03-10T01:43:09.712 INFO:teuthology.orchestra.run.smithi184.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T01:43:09.713 INFO:teuthology.orchestra.run.smithi184.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T01:43:09.713 INFO:teuthology.orchestra.run.smithi184.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T01:43:09.713 INFO:teuthology.orchestra.run.smithi184.stdout: thermald upower usbmuxd 2022-03-10T01:43:09.786 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 13 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T01:43:09.786 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 80.1 MB of archives. 2022-03-10T01:43:09.786 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T01:43:09.786 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T01:43:09.789 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-10T01:43:09.790 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-10T01:43:09.895 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 13 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T01:43:09.895 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 80.1 MB of archives. 2022-03-10T01:43:09.896 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T01:43:09.896 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T01:43:09.933 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2022-03-10T01:43:09.934 INFO:teuthology.orchestra.run.smithi007.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T01:43:09.934 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-modules-5.13.0-35-generic 2022-03-10T01:43:09.934 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-extra-5.13.0-35-generic thermald upower usbmuxd 2022-03-10T01:43:09.935 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2022-03-10T01:43:09.935 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-10T01:43:09.935 INFO:teuthology.orchestra.run.smithi007.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-35-generic 2022-03-10T01:43:09.987 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-10T01:43:09.988 INFO:teuthology.orchestra.run.smithi007.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-10T01:43:09.989 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd6 linux-image-5.13.0-35-generic linux-image-generic-hwe-20.04 2022-03-10T01:43:09.989 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-5.13.0-35-generic linux-modules-extra-5.13.0-35-generic 2022-03-10T01:43:09.989 INFO:teuthology.orchestra.run.smithi007.stdout: thermald upower usbmuxd 2022-03-10T01:43:10.056 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T01:43:10.076 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T01:43:10.078 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T01:43:10.163 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T01:43:10.168 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 13 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T01:43:10.168 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 80.1 MB of archives. 2022-03-10T01:43:10.169 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-10T01:43:10.169 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-10T01:43:10.172 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T01:43:10.177 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T01:43:10.200 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T01:43:10.201 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T01:43:10.283 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T01:43:10.307 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T01:43:10.440 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-10T01:43:10.441 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-10T01:43:10.443 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-10T01:43:10.535 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-10T01:43:10.538 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [18.3 MB] 2022-03-10T01:43:11.107 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T01:43:11.200 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T01:43:11.402 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T01:43:11.484 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T01:43:11.487 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [10.0 MB] 2022-03-10T01:43:11.784 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-35-generic amd64 5.13.0-35.40~20.04.1 [51.1 MB] 2022-03-10T01:43:13.182 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T01:43:13.183 INFO:teuthology.orchestra.run.smithi071.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T01:43:13.183 INFO:teuthology.orchestra.run.smithi071.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T01:43:13.189 INFO:teuthology.orchestra.run.smithi071.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T01:43:13.191 INFO:teuthology.orchestra.run.smithi071.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T01:43:13.313 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T01:43:13.314 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T01:43:13.314 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T01:43:13.320 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T01:43:13.321 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T01:43:13.531 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.35.40~20.04.20 [2,632 B] 2022-03-10T01:43:13.531 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-10T01:43:13.599 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-10T01:43:13.603 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 80.1 MB in 4s (22.4 MB/s) 2022-03-10T01:43:13.605 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-10T01:43:13.607 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-10T01:43:13.723 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 80.1 MB in 4s (22.2 MB/s) 2022-03-10T01:43:13.826 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T01:43:13.951 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T01:43:14.040 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 80.1 MB in 4s (22.2 MB/s) 2022-03-10T01:43:14.211 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-10T01:43:15.292 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T01:43:15.299 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T01:43:15.401 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T01:43:15.406 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T01:43:15.447 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T01:43:15.581 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T01:43:15.706 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-10T01:43:15.710 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-10T01:43:15.825 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T01:43:15.825 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T01:43:15.839 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T01:43:15.883 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T01:43:15.950 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T01:43:15.967 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T01:43:16.009 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T01:43:16.186 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-10T01:43:16.199 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-10T01:43:16.229 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T01:43:16.236 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T01:43:16.242 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T01:43:16.278 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T01:43:16.362 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T01:43:16.375 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T01:43:16.412 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T01:43:16.547 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-10T01:43:16.562 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-10T01:43:16.614 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T01:43:16.627 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T01:43:16.631 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T01:43:16.667 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T01:43:16.756 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T01:43:16.770 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T01:43:16.815 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T01:43:16.961 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-10T01:43:16.972 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-10T01:43:17.009 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T01:43:17.034 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T01:43:17.047 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T01:43:17.101 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T01:43:17.143 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T01:43:17.159 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T01:43:17.201 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T01:43:17.320 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-10T01:43:17.333 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-10T01:43:17.362 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T01:43:17.403 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T01:43:17.418 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:17.453 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:17.562 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T01:43:17.582 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:17.620 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:17.690 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-5.13.0-35-generic. 2022-03-10T01:43:17.704 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../05-linux-modules-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:17.748 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:19.662 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T01:43:19.675 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:19.765 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:19.983 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T01:43:19.997 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:20.063 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-image-5.13.0-35-generic. 2022-03-10T01:43:20.079 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../06-linux-image-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:20.110 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:20.180 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:20.290 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T01:43:20.305 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:20.348 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:20.586 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T01:43:20.600 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:20.644 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:20.716 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-35-generic. 2022-03-10T01:43:20.730 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-35-generic_5.13.0-35.40~20.04.1_amd64.deb ... 2022-03-10T01:43:20.774 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:26.589 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T01:43:26.603 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T01:43:26.646 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T01:43:26.655 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T01:43:26.670 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T01:43:26.686 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-10T01:43:26.696 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.35.40~20.04.20_amd64.deb ... 2022-03-10T01:43:26.738 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T01:43:26.739 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T01:43:26.932 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T01:43:26.949 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T01:43:26.990 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T01:43:27.042 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T01:43:27.057 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T01:43:27.059 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-10T01:43:27.073 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-10T01:43:27.116 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T01:43:27.117 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T01:43:27.379 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package thermald. 2022-03-10T01:43:27.392 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T01:43:27.437 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package thermald. 2022-03-10T01:43:27.444 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package thermald. 2022-03-10T01:43:27.450 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T01:43:27.454 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T01:43:27.458 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-10T01:43:27.487 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T01:43:27.511 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T01:43:27.987 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package upower. 2022-03-10T01:43:28.108 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package upower. 2022-03-10T01:43:28.109 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T01:43:28.110 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T01:43:28.110 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T01:43:28.111 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package upower. 2022-03-10T01:43:28.111 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-10T01:43:28.111 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T01:43:28.114 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-10T01:43:28.436 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T01:43:28.437 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T01:43:28.449 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T01:43:28.454 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T01:43:28.478 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T01:43:28.487 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T01:43:28.559 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package usbmuxd. 2022-03-10T01:43:28.576 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-10T01:43:28.619 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T01:43:28.768 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T01:43:28.797 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T01:43:28.891 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T01:43:28.907 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-10T01:43:28.925 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T01:43:29.040 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-10T01:43:29.042 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T01:43:29.051 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T01:43:29.141 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-10T01:43:29.169 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T01:43:29.178 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T01:43:29.292 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-10T01:43:29.295 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T01:43:29.303 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T01:43:29.430 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T01:43:29.435 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-10T01:43:29.455 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T01:43:29.561 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-10T01:43:29.573 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T01:43:29.597 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T01:43:29.712 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-10T01:43:30.003 INFO:teuthology.orchestra.run.smithi007.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T01:43:30.029 INFO:teuthology.orchestra.run.smithi071.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T01:43:30.049 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T01:43:30.082 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T01:43:30.165 INFO:teuthology.orchestra.run.smithi184.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-10T01:43:30.200 INFO:teuthology.orchestra.run.smithi007.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T01:43:30.214 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T01:43:30.214 INFO:teuthology.orchestra.run.smithi007.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T01:43:30.222 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-10T01:43:30.285 INFO:teuthology.orchestra.run.smithi071.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T01:43:30.299 INFO:teuthology.orchestra.run.smithi071.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T01:43:30.299 INFO:teuthology.orchestra.run.smithi071.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T01:43:30.417 INFO:teuthology.orchestra.run.smithi184.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-10T01:43:30.429 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-10T01:43:30.429 INFO:teuthology.orchestra.run.smithi184.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-10T01:43:30.708 INFO:teuthology.orchestra.run.smithi007.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T01:43:30.808 INFO:teuthology.orchestra.run.smithi071.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T01:43:30.929 INFO:teuthology.orchestra.run.smithi007.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T01:43:30.931 INFO:teuthology.orchestra.run.smithi184.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-10T01:43:30.971 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T01:43:31.049 INFO:teuthology.orchestra.run.smithi071.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T01:43:31.096 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T01:43:31.165 INFO:teuthology.orchestra.run.smithi184.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-10T01:43:31.210 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-10T01:43:31.288 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T01:43:31.289 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T01:43:31.428 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T01:43:31.428 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T01:43:31.543 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T01:43:31.544 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-10T01:43:31.767 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:31.909 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:32.048 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:33.068 INFO:teuthology.orchestra.run.smithi007.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T01:43:33.069 INFO:teuthology.orchestra.run.smithi007.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T01:43:33.172 INFO:teuthology.orchestra.run.smithi071.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T01:43:33.173 INFO:teuthology.orchestra.run.smithi071.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T01:43:33.228 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:33.345 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:33.360 INFO:teuthology.orchestra.run.smithi184.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-35-generic 2022-03-10T01:43:33.360 INFO:teuthology.orchestra.run.smithi184.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-35-generic 2022-03-10T01:43:33.542 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-modules-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:34.605 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:34.680 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:34.919 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-modules-extra-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:35.964 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T01:43:36.007 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T01:43:36.099 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T01:43:36.141 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T01:43:36.245 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.35.40~20.04.20) ... 2022-03-10T01:43:36.396 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-10T01:43:40.713 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:43:41.347 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:43:41.360 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:43:41.373 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T01:43:41.499 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T01:43:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:43:41.971 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T01:43:41.999 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-10T01:43:42.079 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:42.116 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for linux-image-5.13.0-35-generic (5.13.0-35.40~20.04.1) ... 2022-03-10T01:43:42.272 INFO:teuthology.orchestra.run.smithi184.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T01:43:42.272 INFO:teuthology.orchestra.run.smithi184.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:43:42.306 INFO:teuthology.orchestra.run.smithi071.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-10T01:43:42.306 INFO:teuthology.orchestra.run.smithi071.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:44:05.526 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T01:44:05.527 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub' 2022-03-10T01:44:05.535 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T01:44:05.612 INFO:teuthology.orchestra.run.smithi007.stdout:Generating grub configuration file ... 2022-03-10T01:44:05.640 INFO:teuthology.orchestra.run.smithi184.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T01:44:05.641 INFO:teuthology.orchestra.run.smithi184.stdout:Sourcing file `/etc/default/grub' 2022-03-10T01:44:05.650 INFO:teuthology.orchestra.run.smithi184.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T01:44:05.677 INFO:teuthology.orchestra.run.smithi184.stdout:Generating grub configuration file ... 2022-03-10T01:44:05.864 INFO:teuthology.orchestra.run.smithi071.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-10T01:44:05.864 INFO:teuthology.orchestra.run.smithi071.stdout:Sourcing file `/etc/default/grub' 2022-03-10T01:44:05.875 INFO:teuthology.orchestra.run.smithi071.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T01:44:05.900 INFO:teuthology.orchestra.run.smithi071.stdout:Generating grub configuration file ... 2022-03-10T01:44:05.952 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T01:44:05.985 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:44:06.000 INFO:teuthology.orchestra.run.smithi184.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T01:44:06.035 INFO:teuthology.orchestra.run.smithi184.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:44:06.223 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T01:44:06.252 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:44:06.344 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T01:44:06.355 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T01:44:06.455 INFO:teuthology.orchestra.run.smithi184.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T01:44:06.467 INFO:teuthology.orchestra.run.smithi184.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T01:44:06.644 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T01:44:06.656 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T01:44:08.529 INFO:teuthology.orchestra.run.smithi007.stdout:done 2022-03-10T01:44:08.896 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T01:44:08.924 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T01:44:08.924 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2022-03-10T01:44:08.924 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2022-03-10T01:44:08.924 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2022-03-10T01:44:08.925 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 19 2022-03-10T01:44:08.925 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T01:44:08.925 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2022-03-10T01:44:08.925 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta-hwe-5.13 2022-03-10T01:44:08.926 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T01:44:08.926 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T01:44:08.926 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T01:44:08.926 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2022-03-10T01:44:08.926 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2022-03-10T01:44:08.927 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2022-03-10T01:44:08.927 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2022-03-10T01:44:08.927 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T01:44:08.928 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T01:44:08.928 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T01:44:08.928 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-10T01:44:08.928 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T01:44:08.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2022-03-10T01:44:09.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2022-03-10T01:44:09.135 INFO:teuthology.orchestra.run.smithi184.stdout:done 2022-03-10T01:44:09.147 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:44:09.258 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:44:09.351 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:44:09.371 INFO:teuthology.orchestra.run.smithi071.stdout:done 2022-03-10T01:44:09.449 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:44:09.490 DEBUG:teuthology.orchestra.run.smithi184:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T01:44:09.515 INFO:teuthology.orchestra.run.smithi184.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T01:44:09.516 INFO:teuthology.orchestra.run.smithi184.stdout:Status: install ok installed 2022-03-10T01:44:09.516 INFO:teuthology.orchestra.run.smithi184.stdout:Priority: optional 2022-03-10T01:44:09.516 INFO:teuthology.orchestra.run.smithi184.stdout:Section: kernel 2022-03-10T01:44:09.516 INFO:teuthology.orchestra.run.smithi184.stdout:Installed-Size: 19 2022-03-10T01:44:09.517 INFO:teuthology.orchestra.run.smithi184.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T01:44:09.517 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: amd64 2022-03-10T01:44:09.517 INFO:teuthology.orchestra.run.smithi184.stdout:Source: linux-meta-hwe-5.13 2022-03-10T01:44:09.517 INFO:teuthology.orchestra.run.smithi184.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T01:44:09.518 INFO:teuthology.orchestra.run.smithi184.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T01:44:09.518 INFO:teuthology.orchestra.run.smithi184.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T01:44:09.518 INFO:teuthology.orchestra.run.smithi184.stdout:Recommends: thermald 2022-03-10T01:44:09.518 INFO:teuthology.orchestra.run.smithi184.stdout:Description: Generic Linux kernel image 2022-03-10T01:44:09.519 INFO:teuthology.orchestra.run.smithi184.stdout: This package will always depend on the latest generic kernel image 2022-03-10T01:44:09.519 INFO:teuthology.orchestra.run.smithi184.stdout: available. 2022-03-10T01:44:09.519 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T01:44:09.519 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T01:44:09.520 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T01:44:09.520 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-10T01:44:09.520 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T01:44:09.520 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get clean 2022-03-10T01:44:09.590 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt-get update 2022-03-10T01:44:09.729 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:44:09.733 DEBUG:teuthology.orchestra.run.smithi071:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T01:44:09.762 INFO:teuthology.orchestra.run.smithi071.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T01:44:09.762 INFO:teuthology.orchestra.run.smithi071.stdout:Status: install ok installed 2022-03-10T01:44:09.762 INFO:teuthology.orchestra.run.smithi071.stdout:Priority: optional 2022-03-10T01:44:09.763 INFO:teuthology.orchestra.run.smithi071.stdout:Section: kernel 2022-03-10T01:44:09.763 INFO:teuthology.orchestra.run.smithi071.stdout:Installed-Size: 19 2022-03-10T01:44:09.763 INFO:teuthology.orchestra.run.smithi071.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T01:44:09.763 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: amd64 2022-03-10T01:44:09.764 INFO:teuthology.orchestra.run.smithi071.stdout:Source: linux-meta-hwe-5.13 2022-03-10T01:44:09.764 INFO:teuthology.orchestra.run.smithi071.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T01:44:09.764 INFO:teuthology.orchestra.run.smithi071.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T01:44:09.764 INFO:teuthology.orchestra.run.smithi071.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T01:44:09.764 INFO:teuthology.orchestra.run.smithi071.stdout:Recommends: thermald 2022-03-10T01:44:09.765 INFO:teuthology.orchestra.run.smithi071.stdout:Description: Generic Linux kernel image 2022-03-10T01:44:09.765 INFO:teuthology.orchestra.run.smithi071.stdout: This package will always depend on the latest generic kernel image 2022-03-10T01:44:09.765 INFO:teuthology.orchestra.run.smithi071.stdout: available. 2022-03-10T01:44:09.766 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-35-generic 2022-03-10T01:44:09.766 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-10T01:44:09.766 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-10T01:44:09.767 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-10T01:44:09.767 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2022-03-10T01:44:09.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get clean 2022-03-10T01:44:09.845 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:44:09.870 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get update 2022-03-10T01:44:09.926 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:44:10.016 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:44:10.141 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:44:10.142 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:44:10.212 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:44:10.299 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:44:10.721 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-10T01:44:10.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T01:44:10.812 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-10T01:44:10.988 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-10T01:44:10.988 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-10T01:44:11.175 INFO:teuthology.orchestra.run.smithi007.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T01:44:11.175 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T01:44:11.177 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T01:44:11.197 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T01:44:11.197 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2022-03-10T01:44:11.198 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2022-03-10T01:44:11.198 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2022-03-10T01:44:11.198 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 19 2022-03-10T01:44:11.198 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T01:44:11.199 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2022-03-10T01:44:11.199 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta-hwe-5.13 2022-03-10T01:44:11.199 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T01:44:11.199 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T01:44:11.200 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T01:44:11.200 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2022-03-10T01:44:11.200 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2022-03-10T01:44:11.200 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2022-03-10T01:44:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2022-03-10T01:44:11.201 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2022-03-10T01:44:11.244 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.Mz4FyJlh3O 2022-03-10T01:44:11.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Mz4FyJlh3O 2022-03-10T01:44:11.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.Mz4FyJlh3O 2022-03-10T01:44:11.280 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2022-03-10T01:44:11.300 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.Mz4FyJlh3O is 8KB 2022-03-10T01:44:11.304 DEBUG:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T01:44:11.316 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.Mz4FyJlh3O 2022-03-10T01:44:11.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T01:44:11.369 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2022-03-10T01:44:11.377 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:44:11.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T01:44:11.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T01:44:11.414 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T01:44:11.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo update-grub 2022-03-10T01:44:11.524 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2022-03-10T01:44:11.525 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2022-03-10T01:44:11.669 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2022-03-10T01:44:11.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-10T01:44:11.692 INFO:teuthology.orchestra.run.smithi184.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T01:44:11.692 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T01:44:11.694 DEBUG:teuthology.orchestra.run.smithi184:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T01:44:11.716 INFO:teuthology.orchestra.run.smithi184.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T01:44:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:Status: install ok installed 2022-03-10T01:44:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:Priority: optional 2022-03-10T01:44:11.717 INFO:teuthology.orchestra.run.smithi184.stdout:Section: kernel 2022-03-10T01:44:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:Installed-Size: 19 2022-03-10T01:44:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T01:44:11.718 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: amd64 2022-03-10T01:44:11.719 INFO:teuthology.orchestra.run.smithi184.stdout:Source: linux-meta-hwe-5.13 2022-03-10T01:44:11.719 INFO:teuthology.orchestra.run.smithi184.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T01:44:11.719 INFO:teuthology.orchestra.run.smithi184.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T01:44:11.719 INFO:teuthology.orchestra.run.smithi184.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T01:44:11.720 INFO:teuthology.orchestra.run.smithi184.stdout:Recommends: thermald 2022-03-10T01:44:11.720 INFO:teuthology.orchestra.run.smithi184.stdout:Description: Generic Linux kernel image 2022-03-10T01:44:11.720 INFO:teuthology.orchestra.run.smithi184.stdout: This package will always depend on the latest generic kernel image 2022-03-10T01:44:11.721 INFO:teuthology.orchestra.run.smithi184.stdout: available. 2022-03-10T01:44:11.721 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2022-03-10T01:44:11.732 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.foOiRKYwbp 2022-03-10T01:44:11.732 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub/grub.cfg /tmp/tmp.foOiRKYwbp 2022-03-10T01:44:11.755 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2022-03-10T01:44:11.792 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.foOiRKYwbp 2022-03-10T01:44:11.829 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.foOiRKYwbp is 8KB 2022-03-10T01:44:11.843 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.foOiRKYwbp 2022-03-10T01:44:11.849 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T01:44:11.908 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:44:11.908 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T01:44:11.909 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T01:44:11.926 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2022-03-10T01:44:11.926 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2022-03-10T01:44:11.968 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T01:44:11.968 DEBUG:teuthology.orchestra.run.smithi184:> sudo update-grub 2022-03-10T01:44:12.110 INFO:teuthology.orchestra.run.smithi071.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.35.40~20.04.20). 2022-03-10T01:44:12.111 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded. 2022-03-10T01:44:12.112 DEBUG:teuthology.orchestra.run.smithi071:> dpkg -s linux-image-generic-hwe-20.04 2022-03-10T01:44:12.133 INFO:teuthology.orchestra.run.smithi071.stdout:Package: linux-image-generic-hwe-20.04 2022-03-10T01:44:12.133 INFO:teuthology.orchestra.run.smithi071.stdout:Status: install ok installed 2022-03-10T01:44:12.133 INFO:teuthology.orchestra.run.smithi071.stdout:Priority: optional 2022-03-10T01:44:12.133 INFO:teuthology.orchestra.run.smithi071.stdout:Section: kernel 2022-03-10T01:44:12.134 INFO:teuthology.orchestra.run.smithi071.stdout:Installed-Size: 19 2022-03-10T01:44:12.134 INFO:teuthology.orchestra.run.smithi071.stdout:Maintainer: Ubuntu Kernel Team 2022-03-10T01:44:12.134 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: amd64 2022-03-10T01:44:12.134 INFO:teuthology.orchestra.run.smithi071.stdout:Source: linux-meta-hwe-5.13 2022-03-10T01:44:12.135 INFO:teuthology.orchestra.run.smithi071.stdout:Version: 5.13.0.35.40~20.04.20 2022-03-10T01:44:12.135 INFO:teuthology.orchestra.run.smithi071.stdout:Provides: virtualbox-guest-modules (= 5.13.0-35), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-10T01:44:12.135 INFO:teuthology.orchestra.run.smithi071.stdout:Depends: linux-image-5.13.0-35-generic, linux-modules-extra-5.13.0-35-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-10T01:44:12.136 INFO:teuthology.orchestra.run.smithi071.stdout:Recommends: thermald 2022-03-10T01:44:12.136 INFO:teuthology.orchestra.run.smithi071.stdout:Description: Generic Linux kernel image 2022-03-10T01:44:12.136 INFO:teuthology.orchestra.run.smithi071.stdout: This package will always depend on the latest generic kernel image 2022-03-10T01:44:12.136 INFO:teuthology.orchestra.run.smithi071.stdout: available. 2022-03-10T01:44:12.137 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2022-03-10T01:44:12.143 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.OuYnfBLuzu 2022-03-10T01:44:12.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub/grub.cfg /tmp/tmp.OuYnfBLuzu 2022-03-10T01:44:12.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.OuYnfBLuzu 2022-03-10T01:44:12.244 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.OuYnfBLuzu is 8KB 2022-03-10T01:44:12.295 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.OuYnfBLuzu 2022-03-10T01:44:12.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-10T01:44:12.315 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:44:12.316 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-10T01:44:12.316 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-10T01:44:12.352 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-35-generic 2022-03-10T01:44:12.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo update-grub 2022-03-10T01:44:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2022-03-10T01:44:12.641 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T01:44:12.653 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2022-03-10T01:44:12.950 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T01:44:12.963 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:44:13.205 INFO:teuthology.orchestra.run.smithi184.stderr:Sourcing file `/etc/default/grub' 2022-03-10T01:44:13.208 INFO:teuthology.orchestra.run.smithi184.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T01:44:13.225 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2022-03-10T01:44:13.303 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T01:44:13.316 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T01:44:13.480 INFO:teuthology.orchestra.run.smithi184.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T01:44:13.491 INFO:teuthology.orchestra.run.smithi184.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:44:13.639 INFO:teuthology.orchestra.run.smithi071.stderr:Sourcing file `/etc/default/grub' 2022-03-10T01:44:13.640 INFO:teuthology.orchestra.run.smithi071.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-10T01:44:13.648 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2022-03-10T01:44:13.886 INFO:teuthology.orchestra.run.smithi184.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T01:44:13.892 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-5.13.0-35-generic 2022-03-10T01:44:13.897 INFO:teuthology.orchestra.run.smithi184.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T01:44:13.904 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-5.13.0-35-generic 2022-03-10T01:44:14.302 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-10T01:44:14.315 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-10T01:44:14.442 INFO:teuthology.orchestra.run.smithi007.stderr:done 2022-03-10T01:44:14.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2022-03-10T01:44:15.539 INFO:teuthology.orchestra.run.smithi184.stderr:done 2022-03-10T01:44:15.542 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2022-03-10T01:44:15.947 INFO:teuthology.orchestra.run.smithi071.stderr:done 2022-03-10T01:44:15.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2022-03-10T01:44:44.471 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-10T01:44:44.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-10T01:44:44.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:44:45.546 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-10T01:44:45.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2022-03-10T01:44:45.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:44:45.953 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-10T01:44:45.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2022-03-10T01:44:45.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:45:02.923 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2022-03-10T01:45:03.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2022-03-10T01:45:04.463 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2022-03-10T01:45:32.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-10T01:45:32.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:45:33.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2022-03-10T01:45:33.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:45:34.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2022-03-10T01:45:34.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:45:37.003 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2022-03-10T01:45:37.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2022-03-10T01:45:48.726 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-10T01:45:49.648 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-10T01:45:49.649 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-35-generic"... 2022-03-10T01:45:49.649 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2022-03-10T01:45:49.656 INFO:teuthology.orchestra.run.smithi007.stdout:5.13.0-35-generic 2022-03-10T01:45:49.656 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T01:45:49.656 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T01:45:49.657 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T01:45:50.658 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-10T01:45:50.659 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T01:45:50.742 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2022-03-10T01:45:50.766 DEBUG:teuthology.parallel:result is None 2022-03-10T01:46:07.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2022-03-10T01:46:07.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:46:07.203 DEBUG:teuthology.orchestra.run.smithi184:> true 2022-03-10T01:46:07.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2022-03-10T01:46:07.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:46:07.653 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2022-03-10T01:46:07.653 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-35-generic"... 2022-03-10T01:46:07.654 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2022-03-10T01:46:07.703 INFO:teuthology.orchestra.run.smithi184.stdout:5.13.0-35-generic 2022-03-10T01:46:07.703 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T01:46:07.703 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T01:46:07.704 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T01:46:07.770 DEBUG:teuthology.orchestra.run.smithi071:> true 2022-03-10T01:46:08.302 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2022-03-10T01:46:08.302 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-35-generic"... 2022-03-10T01:46:08.303 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2022-03-10T01:46:08.351 INFO:teuthology.orchestra.run.smithi071.stdout:5.13.0-35-generic 2022-03-10T01:46:08.351 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-35-generic vs 5.13.0-35-generic 2022-03-10T01:46:08.352 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-10T01:46:08.352 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-10T01:46:08.705 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-10T01:46:08.705 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T01:46:08.764 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2022-03-10T01:46:08.791 DEBUG:teuthology.parallel:result is None 2022-03-10T01:46:09.353 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-10T01:46:09.353 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-10T01:46:09.413 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2022-03-10T01:46:09.435 DEBUG:teuthology.parallel:result is None 2022-03-10T01:46:09.436 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-10T01:46:09.445 INFO:teuthology.task.internal:Creating test directory... 2022-03-10T01:46:09.446 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T01:46:09.449 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T01:46:09.452 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-10T01:46:09.460 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-10T01:46:09.496 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-10T01:46:09.573 INFO:teuthology.task.internal:Creating archive directory... 2022-03-10T01:46:09.574 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T01:46:09.577 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T01:46:09.580 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-10T01:46:09.618 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-10T01:46:09.715 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-10T01:46:09.716 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T01:46:09.719 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T01:46:09.722 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-10T01:46:09.739 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T01:46:09.742 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T01:46:09.748 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T01:46:09.749 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T01:46:09.751 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T01:46:09.762 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-10T01:46:09.763 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-10T01:46:09.798 INFO:teuthology.task.internal:Configuring sudo... 2022-03-10T01:46:09.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T01:46:09.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T01:46:09.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-10T01:46:09.822 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-10T01:46:09.909 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-10T01:46:09.910 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T01:46:09.913 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T01:46:09.916 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-10T01:46:09.924 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T01:46:09.931 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T01:46:09.983 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:46:09.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T01:46:10.008 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T01:46:10.017 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T01:46:10.022 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:46:10.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T01:46:10.048 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-10T01:46:10.057 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-10T01:46:10.065 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:46:10.066 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-10T01:46:10.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2022-03-10T01:46:10.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2022-03-10T01:46:10.143 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2022-03-10T01:46:10.192 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-10T01:46:10.205 INFO:teuthology.task.internal:Starting timer... 2022-03-10T01:46:10.206 INFO:teuthology.run_tasks:Running task pcp... 2022-03-10T01:46:10.258 INFO:teuthology.run_tasks:Running task selinux... 2022-03-10T01:46:10.342 DEBUG:teuthology.task.selinux:Excluding smithi007: OS 'ubuntu' does not support SELinux 2022-03-10T01:46:10.343 DEBUG:teuthology.task.selinux:Excluding smithi071: OS 'ubuntu' does not support SELinux 2022-03-10T01:46:10.343 DEBUG:teuthology.task.selinux:Excluding smithi184: OS 'ubuntu' does not support SELinux 2022-03-10T01:46:10.343 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-10T01:46:10.343 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-10T01:46:10.344 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-10T01:46:10.344 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-10T01:46:10.425 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-10T01:46:10.425 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-10T01:46:10.449 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-10T01:46:10.451 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi071.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-10T01:49:20.671 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2022-03-10T01:49:20.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-10T01:49:20.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:49:20.728 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-10T01:49:20.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2022-03-10T01:49:20.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:49:20.904 DEBUG:teuthology.orchestra.run.smithi071:> true 2022-03-10T01:49:21.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2022-03-10T01:49:21.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-10T01:49:21.082 DEBUG:teuthology.orchestra.run.smithi184:> true 2022-03-10T01:49:21.201 INFO:teuthology.run_tasks:Running task clock... 2022-03-10T01:49:21.228 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-10T01:49:21.229 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T01:49:21.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T01:49:21.236 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T01:49:21.237 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T01:49:21.240 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-10T01:49:21.241 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-10T01:49:21.269 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T01:49:21.271 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: Command line: ntpd -gq 2022-03-10T01:49:21.272 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: proto: precision = 0.123 usec (-23) 2022-03-10T01:49:21.273 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T01:49:21.274 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: Command line: ntpd -gq 2022-03-10T01:49:21.274 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: proto: precision = 0.082 usec (-23) 2022-03-10T01:49:21.275 INFO:teuthology.orchestra.run.smithi007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T01:49:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T01:49:21.277 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T01:49:21.278 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T01:49:21.279 INFO:teuthology.orchestra.run.smithi007.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T01:49:21.279 INFO:teuthology.orchestra.run.smithi071.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T01:49:21.284 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T01:49:21.284 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T01:49:21.285 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T01:49:21.285 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T01:49:21.286 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: Listen normally on 3 ens1f0 172.21.15.7:123 2022-03-10T01:49:21.286 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: Listen normally on 4 lo [::1]:123 2022-03-10T01:49:21.287 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15ea%4]:123 2022-03-10T01:49:21.287 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:21 ntpd[8940]: Listening on routing socket on fd #22 for interface updates 2022-03-10T01:49:21.288 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T01:49:21.288 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T01:49:21.289 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T01:49:21.289 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T01:49:21.290 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: Listen normally on 3 enp3s0f1 172.21.15.71:123 2022-03-10T01:49:21.290 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: Listen normally on 4 lo [::1]:123 2022-03-10T01:49:21.291 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d765%5]:123 2022-03-10T01:49:21.291 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:21 ntpd[9029]: Listening on routing socket on fd #22 for interface updates 2022-03-10T01:49:21.292 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-10T01:49:21.292 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: Command line: ntpd -gq 2022-03-10T01:49:21.293 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: proto: precision = 0.100 usec (-23) 2022-03-10T01:49:21.294 INFO:teuthology.orchestra.run.smithi184.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T01:49:21.294 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-10T01:49:21.295 INFO:teuthology.orchestra.run.smithi184.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-10T01:49:21.298 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: restrict ::: KOD does nothing without LIMITED. 2022-03-10T01:49:21.298 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: Listen and drop on 0 v6wildcard [::]:123 2022-03-10T01:49:21.298 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-10T01:49:21.299 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: Listen normally on 2 lo 127.0.0.1:123 2022-03-10T01:49:21.300 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: Listen normally on 3 enp3s0f1 172.21.15.184:123 2022-03-10T01:49:21.300 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: Listen normally on 4 lo [::1]:123 2022-03-10T01:49:21.300 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7b8f%5]:123 2022-03-10T01:49:21.301 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:21 ntpd[9029]: Listening on routing socket on fd #22 for interface updates 2022-03-10T01:49:22.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:22.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:22.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:22.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:22.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:22.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:22.276 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:22.276 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:22.277 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:22.277 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:22.278 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:22 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:23.273 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:23.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:23.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:23.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:23.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:23.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:23.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:23.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:23.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:23.277 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:23.277 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:23.278 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:23.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:23.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:23.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:23.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:23.283 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:23 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:23.283 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:23 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:24.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:24.276 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:24 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:24.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:24.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:24.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:24.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:24.282 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:24.282 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:24.282 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:24.282 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:24 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:25.270 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.270 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:25 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:25 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:25 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:25.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:25 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:25.273 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.273 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.273 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:25.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:25.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:25.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:25.277 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:25.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:25.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:25.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:25.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:25.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:25 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:26.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:26.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:26.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:26.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:26.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.276 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:26.276 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:26 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:26.279 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.279 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.279 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:26.280 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:26.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:26.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:26.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.283 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.283 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.283 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:26.283 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:26.283 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:26.284 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:26.284 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:26.284 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:26 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:27.270 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:27 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:27 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:27 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:27 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:27 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:27.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:27 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:27.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.275 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:27.277 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:27.277 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.277 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.277 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:27.278 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:27.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.281 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:27.281 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:27.281 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:27.281 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:27.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:27.282 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:27 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:28.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.271 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.272 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.273 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:28.274 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:28.274 INFO:teuthology.orchestra.run.smithi007.stdout:10 Mar 01:49:28 ntpd[8940]: ntpd: time slew -0.001931 s 2022-03-10T01:49:28.275 INFO:teuthology.orchestra.run.smithi007.stdout:ntpd: time slew -0.001931s 2022-03-10T01:49:28.275 INFO:teuthology.orchestra.run.smithi007.stderr:10 Mar 01:49:28 ntpd[8940]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T01:49:28.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:28.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:28.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.281 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:28.281 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:28.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.281 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.282 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.282 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.282 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.282 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.283 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:28.283 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:28.283 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:28.283 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:28.284 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:28.284 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:28 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:28.335 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T01:49:28.336 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2022-03-10T01:49:28.336 INFO:teuthology.orchestra.run.smithi007.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:28.336 INFO:teuthology.orchestra.run.smithi007.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:28.337 INFO:teuthology.orchestra.run.smithi007.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:28.337 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.272 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:29.273 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:29.273 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:29.273 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:29.273 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:29.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:29.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:29.274 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:29.275 INFO:teuthology.orchestra.run.smithi071.stdout:10 Mar 01:49:29 ntpd[9029]: ntpd: time slew -0.000301 s 2022-03-10T01:49:29.275 INFO:teuthology.orchestra.run.smithi071.stdout:ntpd: time slew -0.000301s 2022-03-10T01:49:29.276 INFO:teuthology.orchestra.run.smithi071.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T01:49:29.277 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:29.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:29.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:29.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:29.278 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:29.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:29.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:29.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:29.279 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-10T01:49:29.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/rawstats.20220310: Permission denied 2022-03-10T01:49:29.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-10T01:49:29.280 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/peerstats.20220310: Permission denied 2022-03-10T01:49:29.280 INFO:teuthology.orchestra.run.smithi184.stdout:10 Mar 01:49:29 ntpd[9029]: ntpd: time slew +0.004749 s 2022-03-10T01:49:29.281 INFO:teuthology.orchestra.run.smithi184.stdout:ntpd: time slew +0.004749s 2022-03-10T01:49:29.281 INFO:teuthology.orchestra.run.smithi184.stderr:10 Mar 01:49:29 ntpd[9029]: can't open /var/log/ntpstats/loopstats.20220310: Permission denied 2022-03-10T01:49:29.338 INFO:teuthology.orchestra.run.smithi071.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T01:49:29.338 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================== 2022-03-10T01:49:29.338 INFO:teuthology.orchestra.run.smithi071.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.339 INFO:teuthology.orchestra.run.smithi071.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.339 INFO:teuthology.orchestra.run.smithi071.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.339 INFO:teuthology.orchestra.run.smithi071.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.341 INFO:teuthology.orchestra.run.smithi184.stdout: remote refid st t when poll reach delay offset jitter 2022-03-10T01:49:29.341 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================== 2022-03-10T01:49:29.342 INFO:teuthology.orchestra.run.smithi184.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.342 INFO:teuthology.orchestra.run.smithi184.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.342 INFO:teuthology.orchestra.run.smithi184.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.343 INFO:teuthology.orchestra.run.smithi184.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-10T01:49:29.344 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-10T01:49:29.361 INFO:tasks.kubeadm:Kubeadm start 2022-03-10T01:49:29.361 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-03-10T01:49:29.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T01:49:29.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T01:49:29.368 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-10T01:49:29.389 INFO:teuthology.orchestra.run.smithi007.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T01:49:29.391 INFO:teuthology.orchestra.run.smithi071.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T01:49:29.394 INFO:teuthology.orchestra.run.smithi184.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-10T01:49:29.397 INFO:teuthology.orchestra.run.smithi007.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T01:49:29.400 INFO:teuthology.orchestra.run.smithi071.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T01:49:29.401 INFO:teuthology.orchestra.run.smithi184.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-10T01:49:29.403 INFO:teuthology.orchestra.run.smithi007.stdout:net.ipv4.ip_forward = 1 2022-03-10T01:49:29.410 INFO:teuthology.orchestra.run.smithi071.stdout:net.ipv4.ip_forward = 1 2022-03-10T01:49:29.411 INFO:teuthology.orchestra.run.smithi184.stdout:net.ipv4.ip_forward = 1 2022-03-10T01:49:29.484 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:49:29.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T01:49:29.500 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:49:29.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/docker/daemon.json 2022-03-10T01:49:29.559 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:49:29.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T01:49:29.576 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:49:29.577 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/docker/daemon.json 2022-03-10T01:49:29.591 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:49:29.591 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-10T01:49:29.610 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:49:29.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/docker/daemon.json 2022-03-10T01:49:29.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl restart docker || true 2022-03-10T01:49:29.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl restart docker || true 2022-03-10T01:49:29.673 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl restart docker || true 2022-03-10T01:49:30.675 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-10T01:49:30.682 INFO:tasks.kubeadm:172.21.15.7 is in 172.21.0.0/20 at pos 3846 2022-03-10T01:49:30.695 INFO:tasks.kubeadm:Pod subnet: 10.248.48.0/21 2022-03-10T01:49:30.695 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-10T01:49:30.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T01:49:30.698 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T01:49:30.702 DEBUG:teuthology.orchestra.run.smithi184:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-10T01:49:30.721 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-10T01:49:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:30.722 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-10T01:49:30.725 INFO:teuthology.orchestra.run.smithi071.stderr: 2022-03-10T01:49:30.725 INFO:teuthology.orchestra.run.smithi071.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:30.726 INFO:teuthology.orchestra.run.smithi071.stderr: 2022-03-10T01:49:30.729 INFO:teuthology.orchestra.run.smithi184.stderr: 2022-03-10T01:49:30.729 INFO:teuthology.orchestra.run.smithi184.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:30.730 INFO:teuthology.orchestra.run.smithi184.stderr: 2022-03-10T01:49:30.858 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:49:30.860 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:49:30.954 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:49:30.960 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:49:30.978 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:49:30.978 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:49:31.042 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:49:31.044 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:49:31.049 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:49:31.124 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:49:31.126 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:49:31.133 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:49:32.319 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2022-03-10T01:49:32.398 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-10T01:49:32.469 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2022-03-10T01:49:32.490 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2022-03-10T01:49:32.490 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2022-03-10T01:49:32.575 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-10T01:49:32.575 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-10T01:49:32.579 INFO:teuthology.orchestra.run.smithi071.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T01:49:32.593 INFO:teuthology.orchestra.run.smithi071.stderr: 2022-03-10T01:49:32.594 INFO:teuthology.orchestra.run.smithi071.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:32.594 INFO:teuthology.orchestra.run.smithi071.stderr: 2022-03-10T01:49:32.650 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2022-03-10T01:49:32.653 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2022-03-10T01:49:32.653 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2022-03-10T01:49:32.672 INFO:teuthology.orchestra.run.smithi007.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T01:49:32.686 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-10T01:49:32.687 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:32.687 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-10T01:49:32.732 INFO:teuthology.orchestra.run.smithi184.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T01:49:32.741 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-10T01:49:32.744 INFO:teuthology.orchestra.run.smithi184.stderr: 2022-03-10T01:49:32.745 INFO:teuthology.orchestra.run.smithi184.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:32.745 INFO:teuthology.orchestra.run.smithi184.stderr: 2022-03-10T01:49:32.798 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2022-03-10T01:49:32.802 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2022-03-10T01:49:32.803 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2022-03-10T01:49:32.918 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-10T01:49:32.919 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-10T01:49:32.956 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2022-03-10T01:49:32.957 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2022-03-10T01:49:32.967 INFO:teuthology.orchestra.run.smithi071.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T01:49:32.967 INFO:teuthology.orchestra.run.smithi071.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T01:49:32.967 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2022-03-10T01:49:32.968 INFO:teuthology.orchestra.run.smithi071.stdout: apt-transport-https 2022-03-10T01:49:33.103 INFO:teuthology.orchestra.run.smithi007.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T01:49:33.104 INFO:teuthology.orchestra.run.smithi007.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T01:49:33.104 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-10T01:49:33.104 INFO:teuthology.orchestra.run.smithi007.stdout: apt-transport-https 2022-03-10T01:49:33.123 INFO:teuthology.orchestra.run.smithi184.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-10T01:49:33.123 INFO:teuthology.orchestra.run.smithi184.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-10T01:49:33.123 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2022-03-10T01:49:33.124 INFO:teuthology.orchestra.run.smithi184.stdout: apt-transport-https 2022-03-10T01:49:33.154 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 1 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:49:33.155 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 4,680 B of archives. 2022-03-10T01:49:33.155 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T01:49:33.155 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T01:49:33.286 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 1 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:49:33.286 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 4,680 B of archives. 2022-03-10T01:49:33.286 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T01:49:33.287 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T01:49:33.309 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 1 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:49:33.310 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 4,680 B of archives. 2022-03-10T01:49:33.310 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-10T01:49:33.310 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-10T01:49:33.372 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T01:49:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T01:49:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: falling back to frontend: Readline 2022-03-10T01:49:33.392 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T01:49:33.393 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T01:49:33.393 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: falling back to frontend: Teletype 2022-03-10T01:49:33.398 INFO:teuthology.orchestra.run.smithi071.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T01:49:33.445 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 4,680 B in 0s (26.3 kB/s) 2022-03-10T01:49:33.518 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T01:49:33.519 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T01:49:33.519 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Readline 2022-03-10T01:49:33.531 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T01:49:33.533 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T01:49:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T01:49:33.534 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Teletype 2022-03-10T01:49:33.535 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T01:49:33.535 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T01:49:33.535 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: falling back to frontend: Readline 2022-03-10T01:49:33.539 INFO:teuthology.orchestra.run.smithi007.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T01:49:33.553 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T01:49:33.554 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T01:49:33.554 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: falling back to frontend: Teletype 2022-03-10T01:49:33.558 INFO:teuthology.orchestra.run.smithi184.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T01:49:33.575 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T01:49:33.578 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T01:49:33.590 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 4,680 B in 0s (26.8 kB/s) 2022-03-10T01:49:33.606 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 4,680 B in 0s (26.5 kB/s) 2022-03-10T01:49:33.613 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T01:49:33.690 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T01:49:33.701 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package apt-transport-https. 2022-03-10T01:49:33.732 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T01:49:33.735 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T01:49:33.745 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-10T01:49:33.748 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-10T01:49:33.772 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T01:49:33.791 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-10T01:49:33.864 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T01:49:34.056 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T01:49:34.084 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-10T01:49:34.406 INFO:teuthology.orchestra.run.smithi071.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T01:49:34.421 INFO:teuthology.orchestra.run.smithi071.stderr: 2022-03-10T01:49:34.422 INFO:teuthology.orchestra.run.smithi071.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:34.422 INFO:teuthology.orchestra.run.smithi071.stderr: 2022-03-10T01:49:34.555 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:49:34.577 INFO:teuthology.orchestra.run.smithi184.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T01:49:34.593 INFO:teuthology.orchestra.run.smithi184.stderr: 2022-03-10T01:49:34.593 INFO:teuthology.orchestra.run.smithi184.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:34.594 INFO:teuthology.orchestra.run.smithi184.stderr: 2022-03-10T01:49:34.595 INFO:teuthology.orchestra.run.smithi007.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-10T01:49:34.611 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-10T01:49:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:34.619 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-10T01:49:34.669 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:49:34.724 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:49:34.752 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:49:34.835 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:49:34.835 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:49:34.871 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-10T01:49:34.871 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-10T01:49:34.916 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:49:34.941 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-10T01:49:34.999 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:49:35.023 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-10T01:49:36.908 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T01:49:37.024 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T01:49:37.164 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 63.3 kB in 3s (24.4 kB/s) 2022-03-10T01:49:38.015 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2022-03-10T01:49:38.188 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2022-03-10T01:49:38.189 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2022-03-10T01:49:38.280 INFO:teuthology.orchestra.run.smithi071.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T01:49:38.297 INFO:teuthology.orchestra.run.smithi071.stderr: 2022-03-10T01:49:38.297 INFO:teuthology.orchestra.run.smithi071.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:38.298 INFO:teuthology.orchestra.run.smithi071.stderr: 2022-03-10T01:49:38.348 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2022-03-10T01:49:38.480 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T01:49:38.521 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2022-03-10T01:49:38.522 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2022-03-10T01:49:38.604 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T01:49:38.662 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-10T01:49:38.670 INFO:teuthology.orchestra.run.smithi071.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T01:49:38.671 INFO:teuthology.orchestra.run.smithi071.stdout:bridge-utils set to manually installed. 2022-03-10T01:49:38.671 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2022-03-10T01:49:38.672 INFO:teuthology.orchestra.run.smithi071.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T01:49:38.672 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2022-03-10T01:49:38.672 INFO:teuthology.orchestra.run.smithi071.stdout: nftables 2022-03-10T01:49:38.720 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2022-03-10T01:49:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T01:49:38.741 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 63.3 kB in 4s (15.9 kB/s) 2022-03-10T01:49:38.780 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-10T01:49:38.921 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 8 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:49:38.922 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 73.6 MB of archives. 2022-03-10T01:49:38.922 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T01:49:38.922 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T01:49:38.923 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 63.3 kB in 4s (15.1 kB/s) 2022-03-10T01:49:38.971 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T01:49:39.110 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T01:49:39.241 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T01:49:39.356 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T01:49:39.611 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-10T01:49:39.765 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2022-03-10T01:49:39.788 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-10T01:49:39.789 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-10T01:49:39.885 INFO:teuthology.orchestra.run.smithi007.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T01:49:39.900 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-10T01:49:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:39.900 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-10T01:49:39.939 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2022-03-10T01:49:39.940 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2022-03-10T01:49:39.956 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-10T01:49:40.033 INFO:teuthology.orchestra.run.smithi184.stdout:87 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-10T01:49:40.046 INFO:teuthology.orchestra.run.smithi184.stderr: 2022-03-10T01:49:40.047 INFO:teuthology.orchestra.run.smithi184.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-10T01:49:40.047 INFO:teuthology.orchestra.run.smithi184.stderr: 2022-03-10T01:49:40.102 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2022-03-10T01:49:40.125 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-10T01:49:40.125 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-10T01:49:40.269 INFO:teuthology.orchestra.run.smithi007.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T01:49:40.269 INFO:teuthology.orchestra.run.smithi007.stdout:bridge-utils set to manually installed. 2022-03-10T01:49:40.270 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2022-03-10T01:49:40.271 INFO:teuthology.orchestra.run.smithi007.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T01:49:40.271 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2022-03-10T01:49:40.272 INFO:teuthology.orchestra.run.smithi007.stdout: nftables 2022-03-10T01:49:40.279 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2022-03-10T01:49:40.280 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2022-03-10T01:49:40.321 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-10T01:49:40.322 INFO:teuthology.orchestra.run.smithi007.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T01:49:40.330 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T01:49:40.421 INFO:teuthology.orchestra.run.smithi184.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-10T01:49:40.421 INFO:teuthology.orchestra.run.smithi184.stdout:bridge-utils set to manually installed. 2022-03-10T01:49:40.421 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2022-03-10T01:49:40.422 INFO:teuthology.orchestra.run.smithi184.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-10T01:49:40.422 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2022-03-10T01:49:40.423 INFO:teuthology.orchestra.run.smithi184.stdout: nftables 2022-03-10T01:49:40.472 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2022-03-10T01:49:40.473 INFO:teuthology.orchestra.run.smithi184.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-10T01:49:40.583 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 8 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:49:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 73.6 MB of archives. 2022-03-10T01:49:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T01:49:40.585 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T01:49:40.733 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 8 newly installed, 0 to remove and 87 not upgraded. 2022-03-10T01:49:40.734 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 73.6 MB of archives. 2022-03-10T01:49:40.734 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-10T01:49:40.734 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-10T01:49:40.778 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T01:49:40.917 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T01:49:40.929 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-10T01:49:40.948 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-10T01:49:41.118 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T01:49:41.246 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T01:49:41.326 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-10T01:49:41.381 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T01:49:41.723 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-10T01:49:41.885 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T01:49:42.075 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T01:49:42.597 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-10T01:49:42.631 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T01:49:42.747 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T01:49:42.748 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T01:49:42.748 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: falling back to frontend: Readline 2022-03-10T01:49:42.756 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T01:49:42.757 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T01:49:42.757 INFO:teuthology.orchestra.run.smithi071.stderr:debconf: falling back to frontend: Teletype 2022-03-10T01:49:42.761 INFO:teuthology.orchestra.run.smithi071.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T01:49:42.803 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T01:49:42.812 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 73.6 MB in 4s (19.4 MB/s) 2022-03-10T01:49:42.893 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package conntrack. 2022-03-10T01:49:42.936 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T01:49:42.939 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T01:49:42.983 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T01:49:43.206 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-10T01:49:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T01:49:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T01:49:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Readline 2022-03-10T01:49:43.277 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package cri-tools. 2022-03-10T01:49:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T01:49:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T01:49:43.282 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Teletype 2022-03-10T01:49:43.286 INFO:teuthology.orchestra.run.smithi007.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T01:49:43.293 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T01:49:43.338 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 73.6 MB in 3s (27.1 MB/s) 2022-03-10T01:49:43.339 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T01:49:43.346 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-10T01:49:43.432 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package conntrack. 2022-03-10T01:49:43.478 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T01:49:43.481 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T01:49:43.522 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T01:49:43.775 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: unable to initialize frontend: Dialog 2022-03-10T01:49:43.775 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-10T01:49:43.776 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: falling back to frontend: Readline 2022-03-10T01:49:43.784 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: unable to initialize frontend: Readline 2022-03-10T01:49:43.785 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-10T01:49:43.785 INFO:teuthology.orchestra.run.smithi184.stderr:debconf: falling back to frontend: Teletype 2022-03-10T01:49:43.790 INFO:teuthology.orchestra.run.smithi184.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-10T01:49:43.817 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cri-tools. 2022-03-10T01:49:43.831 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T01:49:43.836 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 73.6 MB in 3s (24.0 MB/s) 2022-03-10T01:49:43.869 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T01:49:43.934 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package conntrack. 2022-03-10T01:49:43.979 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-10T01:49:43.982 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-10T01:49:44.016 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-10T01:49:44.336 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cri-tools. 2022-03-10T01:49:44.349 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-10T01:49:44.387 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-10T01:49:44.391 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ebtables. 2022-03-10T01:49:44.404 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T01:49:44.458 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T01:49:44.716 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T01:49:44.729 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T01:49:44.766 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T01:49:44.899 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ebtables. 2022-03-10T01:49:44.912 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T01:49:44.991 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T01:49:45.245 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T01:49:45.258 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T01:49:45.303 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T01:49:45.451 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ebtables. 2022-03-10T01:49:45.460 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-10T01:49:45.514 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-10T01:49:45.771 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-10T01:49:45.784 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-10T01:49:45.821 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-10T01:49:46.227 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package socat. 2022-03-10T01:49:46.239 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T01:49:46.277 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T01:49:46.664 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package kubelet. 2022-03-10T01:49:46.677 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T01:49:46.714 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T01:49:46.763 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package socat. 2022-03-10T01:49:46.777 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T01:49:46.822 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T01:49:47.193 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubelet. 2022-03-10T01:49:47.205 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T01:49:47.242 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T01:49:47.315 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package socat. 2022-03-10T01:49:47.328 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-10T01:49:47.390 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-10T01:49:47.777 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package kubelet. 2022-03-10T01:49:47.790 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-10T01:49:47.827 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-10T01:49:49.091 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package kubectl. 2022-03-10T01:49:49.109 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T01:49:49.149 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T01:49:49.694 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubectl. 2022-03-10T01:49:49.707 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T01:49:49.744 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T01:49:50.221 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package kubectl. 2022-03-10T01:49:50.233 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-10T01:49:50.271 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-10T01:49:50.368 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package kubeadm. 2022-03-10T01:49:50.383 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T01:49:50.419 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T01:49:50.972 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubeadm. 2022-03-10T01:49:50.985 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T01:49:51.021 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T01:49:51.490 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package kubeadm. 2022-03-10T01:49:51.504 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-10T01:49:51.548 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-10T01:49:51.551 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T01:49:51.672 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T01:49:51.799 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T01:49:52.200 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T01:49:52.308 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T01:49:52.411 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T01:49:52.626 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T01:49:52.686 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-10T01:49:52.752 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T01:49:52.810 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-10T01:49:52.887 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T01:49:52.927 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-10T01:49:53.013 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T01:49:53.196 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T01:49:53.237 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T01:49:53.321 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T01:49:53.451 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T01:49:53.590 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T01:49:53.680 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-10T01:49:53.731 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T01:49:53.806 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-10T01:49:53.810 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T01:49:53.957 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-10T01:49:54.083 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-10T01:49:54.225 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:49:54.304 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-10T01:49:54.316 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T01:49:54.703 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:49:54.809 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-10T01:49:55.204 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-10T01:49:56.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T01:49:56.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T01:49:56.107 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-10T01:50:08.603 INFO:teuthology.orchestra.run.smithi071.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T01:50:09.241 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T01:50:09.794 INFO:teuthology.orchestra.run.smithi184.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-10T01:50:11.932 INFO:teuthology.orchestra.run.smithi071.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T01:50:12.293 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T01:50:12.813 INFO:teuthology.orchestra.run.smithi184.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-10T01:50:14.227 INFO:teuthology.orchestra.run.smithi071.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T01:50:14.639 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T01:50:15.427 INFO:teuthology.orchestra.run.smithi184.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-10T01:50:18.815 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T01:50:19.109 INFO:teuthology.orchestra.run.smithi071.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T01:50:20.001 INFO:teuthology.orchestra.run.smithi184.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-10T01:50:20.565 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T01:50:20.874 INFO:teuthology.orchestra.run.smithi071.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T01:50:21.717 INFO:teuthology.orchestra.run.smithi184.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-10T01:50:27.955 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T01:50:28.791 INFO:teuthology.orchestra.run.smithi184.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T01:50:28.799 INFO:teuthology.orchestra.run.smithi071.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-10T01:50:30.879 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T01:50:31.458 INFO:teuthology.orchestra.run.smithi071.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T01:50:31.691 INFO:teuthology.orchestra.run.smithi184.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-10T01:50:31.696 INFO:tasks.kubeadm:Token: abcdef.0djcvnntlzl1z7nk 2022-03-10T01:50:31.696 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): '172.21.15.7', Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): '172.21.15.71', Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): '172.21.15.184'} 2022-03-10T01:50:31.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo kubeadm init --node-name smithi007 --token abcdef.0djcvnntlzl1z7nk --pod-network-cidr 10.248.48.0/21 2022-03-10T01:50:32.013 INFO:teuthology.orchestra.run.smithi007.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-10T01:50:32.014 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] Running pre-flight checks 2022-03-10T01:50:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-10T01:50:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-10T01:50:32.441 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-10T01:50:32.753 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-10T01:50:32.870 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "ca" certificate and key 2022-03-10T01:50:32.939 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver" certificate and key 2022-03-10T01:50:32.940 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi007] and IPs [10.96.0.1 172.21.15.7] 2022-03-10T01:50:33.081 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-10T01:50:33.264 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-10T01:50:33.537 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-10T01:50:33.722 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-10T01:50:33.976 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-10T01:50:33.977 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi007] and IPs [172.21.15.7 127.0.0.1 ::1] 2022-03-10T01:50:34.217 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-10T01:50:34.218 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi007] and IPs [172.21.15.7 127.0.0.1 ::1] 2022-03-10T01:50:34.347 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-10T01:50:34.463 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-10T01:50:34.599 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "sa" key and public key 2022-03-10T01:50:34.599 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-10T01:50:34.921 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-10T01:50:35.244 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-10T01:50:35.789 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-10T01:50:35.897 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-10T01:50:35.909 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T01:50:35.911 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T01:50:35.911 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Starting the kubelet 2022-03-10T01:50:36.129 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-10T01:50:36.129 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-10T01:50:36.134 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-10T01:50:36.136 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-10T01:50:36.137 INFO:teuthology.orchestra.run.smithi007.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-10T01:50:36.140 INFO:teuthology.orchestra.run.smithi007.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-10T01:50:53.644 INFO:teuthology.orchestra.run.smithi007.stdout:[apiclient] All control plane components are healthy after 17.502973 seconds 2022-03-10T01:50:53.644 INFO:teuthology.orchestra.run.smithi007.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-10T01:50:53.685 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-10T01:50:53.686 INFO:teuthology.orchestra.run.smithi007.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-10T01:50:54.219 INFO:teuthology.orchestra.run.smithi007.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-10T01:50:54.219 INFO:teuthology.orchestra.run.smithi007.stdout:[mark-control-plane] Marking the node smithi007 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-10T01:50:54.219 INFO:teuthology.orchestra.run.smithi007.stdout:[mark-control-plane] Marking the node smithi007 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-10T01:50:54.728 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Using token: abcdef.0djcvnntlzl1z7nk 2022-03-10T01:50:54.728 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-10T01:50:54.747 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-10T01:50:54.773 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-10T01:50:54.781 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-10T01:50:54.789 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-10T01:50:54.798 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-10T01:50:54.863 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-10T01:50:55.307 INFO:teuthology.orchestra.run.smithi007.stdout:[addons] Applied essential addon: CoreDNS 2022-03-10T01:50:55.456 INFO:teuthology.orchestra.run.smithi007.stdout:[addons] Applied essential addon: kube-proxy 2022-03-10T01:50:55.457 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:50:55.457 INFO:teuthology.orchestra.run.smithi007.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-10T01:50:55.457 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:50:55.458 INFO:teuthology.orchestra.run.smithi007.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-10T01:50:55.458 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:50:55.458 INFO:teuthology.orchestra.run.smithi007.stdout: mkdir -p $HOME/.kube 2022-03-10T01:50:55.458 INFO:teuthology.orchestra.run.smithi007.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-10T01:50:55.459 INFO:teuthology.orchestra.run.smithi007.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-10T01:50:55.459 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:50:55.459 INFO:teuthology.orchestra.run.smithi007.stdout:Alternatively, if you are the root user, you can run: 2022-03-10T01:50:55.459 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:50:55.459 INFO:teuthology.orchestra.run.smithi007.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-10T01:50:55.460 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:50:55.460 INFO:teuthology.orchestra.run.smithi007.stdout:You should now deploy a pod network to the cluster. 2022-03-10T01:50:55.460 INFO:teuthology.orchestra.run.smithi007.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-10T01:50:55.460 INFO:teuthology.orchestra.run.smithi007.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-10T01:50:55.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:50:55.461 INFO:teuthology.orchestra.run.smithi007.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-10T01:50:55.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:50:55.461 INFO:teuthology.orchestra.run.smithi007.stdout:kubeadm join 172.21.15.7:6443 --token abcdef.0djcvnntlzl1z7nk \ 2022-03-10T01:50:55.461 INFO:teuthology.orchestra.run.smithi007.stdout: --discovery-token-ca-cert-hash sha256:ef48bb4217a64911495eac7d38ccb59df3631784ec3c2d3aafa7084c9f25ea97 2022-03-10T01:50:55.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo kubeadm join 172.21.15.7:6443 --node-name smithi071 --token abcdef.0djcvnntlzl1z7nk --discovery-token-unsafe-skip-ca-verification 2022-03-10T01:50:55.466 DEBUG:teuthology.orchestra.run.smithi184:> sudo kubeadm join 172.21.15.7:6443 --node-name smithi184 --token abcdef.0djcvnntlzl1z7nk --discovery-token-unsafe-skip-ca-verification 2022-03-10T01:50:55.506 INFO:teuthology.orchestra.run.smithi184.stdout:[preflight] Running pre-flight checks 2022-03-10T01:50:55.507 INFO:teuthology.orchestra.run.smithi071.stdout:[preflight] Running pre-flight checks 2022-03-10T01:51:14.309 INFO:teuthology.orchestra.run.smithi184.stdout:[preflight] Reading configuration from the cluster... 2022-03-10T01:51:14.310 INFO:teuthology.orchestra.run.smithi184.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-10T01:51:14.310 INFO:teuthology.orchestra.run.smithi071.stdout:[preflight] Reading configuration from the cluster... 2022-03-10T01:51:14.310 INFO:teuthology.orchestra.run.smithi071.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-10T01:51:14.659 INFO:teuthology.orchestra.run.smithi184.stderr:W0310 01:51:14.677219 10623 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-10T01:51:14.661 INFO:teuthology.orchestra.run.smithi071.stderr:W0310 01:51:14.672403 10685 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-10T01:51:14.677 INFO:teuthology.orchestra.run.smithi184.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T01:51:14.677 INFO:teuthology.orchestra.run.smithi184.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T01:51:14.677 INFO:teuthology.orchestra.run.smithi184.stdout:[kubelet-start] Starting the kubelet 2022-03-10T01:51:14.678 INFO:teuthology.orchestra.run.smithi071.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-10T01:51:14.679 INFO:teuthology.orchestra.run.smithi071.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-10T01:51:14.679 INFO:teuthology.orchestra.run.smithi071.stdout:[kubelet-start] Starting the kubelet 2022-03-10T01:51:14.892 INFO:teuthology.orchestra.run.smithi071.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-10T01:51:14.897 INFO:teuthology.orchestra.run.smithi184.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-10T01:51:20.426 INFO:teuthology.orchestra.run.smithi184.stdout: 2022-03-10T01:51:20.427 INFO:teuthology.orchestra.run.smithi184.stdout:This node has joined the cluster: 2022-03-10T01:51:20.427 INFO:teuthology.orchestra.run.smithi184.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-10T01:51:20.427 INFO:teuthology.orchestra.run.smithi184.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-10T01:51:20.427 INFO:teuthology.orchestra.run.smithi184.stdout: 2022-03-10T01:51:20.428 INFO:teuthology.orchestra.run.smithi184.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-10T01:51:20.428 INFO:teuthology.orchestra.run.smithi184.stdout: 2022-03-10T01:51:28.416 INFO:teuthology.orchestra.run.smithi071.stdout: 2022-03-10T01:51:28.417 INFO:teuthology.orchestra.run.smithi071.stdout:This node has joined the cluster: 2022-03-10T01:51:28.417 INFO:teuthology.orchestra.run.smithi071.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-10T01:51:28.417 INFO:teuthology.orchestra.run.smithi071.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-10T01:51:28.417 INFO:teuthology.orchestra.run.smithi071.stdout: 2022-03-10T01:51:28.418 INFO:teuthology.orchestra.run.smithi071.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-10T01:51:28.418 INFO:teuthology.orchestra.run.smithi071.stdout: 2022-03-10T01:51:28.423 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:28.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-10T01:51:28.440 INFO:tasks.kubeadm:Setting up kubectl 2022-03-10T01:51:28.441 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T01:51:28.499 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T01:51:28.519 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-10T01:51:28.537 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:28.538 DEBUG:teuthology.orchestra.run.smithi007:> dd of=.kube/config 2022-03-10T01:51:28.546 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:28.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/root/.kube/config 2022-03-10T01:51:28.603 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:51:28.604 DEBUG:teuthology.orchestra.run.smithi071:> dd of=.kube/config 2022-03-10T01:51:28.612 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:51:28.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/root/.kube/config 2022-03-10T01:51:28.673 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:51:28.673 DEBUG:teuthology.orchestra.run.smithi184:> dd of=.kube/config 2022-03-10T01:51:28.681 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:51:28.681 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/root/.kube/config 2022-03-10T01:51:28.696 DEBUG:teuthology.orchestra.run.smithi007:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-03-10T01:51:28.712 INFO:teuthology.orchestra.run.smithi007.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-03-10T01:51:28.712 INFO:teuthology.orchestra.run.smithi007.stderr: Dload Upload Total Spent Left Speed 2022-03-10T01:51:28.808 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-10T01:51:28.809 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: policy/v1beta1 2022-03-10T01:51:28.809 INFO:teuthology.orchestra.run.smithi007.stdout:kind: PodSecurityPolicy 2022-03-10T01:51:28.809 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-10T01:51:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: name: psp.flannel.unprivileged 2022-03-10T01:51:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: annotations: 2022-03-10T01:51:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-03-10T01:51:28.810 INFO:teuthology.orchestra.run.smithi007.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-03-10T01:51:28.811 INFO:teuthology.orchestra.run.smithi007.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-03-10T01:51:28.811 INFO:teuthology.orchestra.run.smithi007.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-03-10T01:51:28.811 INFO:teuthology.orchestra.run.smithi007.stdout:spec: 2022-03-10T01:51:28.811 INFO:teuthology.orchestra.run.smithi007.stdout: privileged: false 2022-03-10T01:51:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: volumes: 2022-03-10T01:51:28.812 INFO:teuthology.orchestra.run.smithi007.stdout: - configMap 2022-03-10T01:51:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: - secret 2022-03-10T01:51:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: - emptyDir 2022-03-10T01:51:28.813 INFO:teuthology.orchestra.run.smithi007.stdout: - hostPath 2022-03-10T01:51:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: allowedHostPaths: 2022-03-10T01:51:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: - pathPrefix: "/etc/cni/net.d" 2022-03-10T01:51:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: - pathPrefix: "/etc/kube-flannel" 2022-03-10T01:51:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: - pathPrefix: "/run/flannel" 2022-03-10T01:51:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: readOnlyRootFilesystem: false 2022-03-10T01:51:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: # Users and groups 2022-03-10T01:51:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: runAsUser: 2022-03-10T01:51:28.815 INFO:teuthology.orchestra.run.smithi007.stdout: rule: RunAsAny 2022-03-10T01:51:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: supplementalGroups: 2022-03-10T01:51:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: rule: RunAsAny 2022-03-10T01:51:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: fsGroup: 2022-03-10T01:51:28.816 INFO:teuthology.orchestra.run.smithi007.stdout: rule: RunAsAny 2022-03-10T01:51:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: # Privilege Escalation 2022-03-10T01:51:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: allowPrivilegeEscalation: false 2022-03-10T01:51:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: defaultAllowPrivilegeEscalation: false 2022-03-10T01:51:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: # Capabilities 2022-03-10T01:51:28.817 INFO:teuthology.orchestra.run.smithi007.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-03-10T01:51:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: defaultAddCapabilities: [] 2022-03-10T01:51:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: requiredDropCapabilities: [] 2022-03-10T01:51:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: # Host namespaces 2022-03-10T01:51:28.818 INFO:teuthology.orchestra.run.smithi007.stdout: hostPID: false 2022-03-10T01:51:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: hostIPC: false 2022-03-10T01:51:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: hostNetwork: true 2022-03-10T01:51:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: hostPorts: 2022-03-10T01:51:28.819 INFO:teuthology.orchestra.run.smithi007.stdout: - min: 0 2022-03-10T01:51:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: max: 65535 2022-03-10T01:51:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: # SELinux 2022-03-10T01:51:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: seLinux: 2022-03-10T01:51:28.820 INFO:teuthology.orchestra.run.smithi007.stdout: # SELinux is unused in CaaSP 2022-03-10T01:51:28.821 INFO:teuthology.orchestra.run.smithi007.stdout: rule: 'RunAsAny' 2022-03-10T01:51:28.821 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-10T01:51:28.821 INFO:teuthology.orchestra.run.smithi007.stdout:kind: ClusterRole 2022-03-10T01:51:28.821 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-10T01:51:28.821 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-10T01:51:28.822 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-10T01:51:28.822 INFO:teuthology.orchestra.run.smithi007.stdout:rules: 2022-03-10T01:51:28.822 INFO:teuthology.orchestra.run.smithi007.stdout:- apiGroups: ['extensions'] 2022-03-10T01:51:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: resources: ['podsecuritypolicies'] 2022-03-10T01:51:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: verbs: ['use'] 2022-03-10T01:51:28.823 INFO:teuthology.orchestra.run.smithi007.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-03-10T01:51:28.824 INFO:teuthology.orchestra.run.smithi007.stdout:- apiGroups: 2022-03-10T01:51:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: - "" 2022-03-10T01:51:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: resources: 2022-03-10T01:51:28.824 INFO:teuthology.orchestra.run.smithi007.stdout: - pods 2022-03-10T01:51:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: verbs: 2022-03-10T01:51:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: - get 2022-03-10T01:51:28.825 INFO:teuthology.orchestra.run.smithi007.stdout:- apiGroups: 2022-03-10T01:51:28.825 INFO:teuthology.orchestra.run.smithi007.stdout: - "" 2022-03-10T01:51:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: resources: 2022-03-10T01:51:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: - nodes 2022-03-10T01:51:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: verbs: 2022-03-10T01:51:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: - list 2022-03-10T01:51:28.826 INFO:teuthology.orchestra.run.smithi007.stdout: - watch 2022-03-10T01:51:28.827 INFO:teuthology.orchestra.run.smithi007.stdout:- apiGroups: 2022-03-10T01:51:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: - "" 2022-03-10T01:51:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: resources: 2022-03-10T01:51:28.827 INFO:teuthology.orchestra.run.smithi007.stdout: - nodes/status 2022-03-10T01:51:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: verbs: 2022-03-10T01:51:28.828 INFO:teuthology.orchestra.run.smithi007.stdout: - patch 2022-03-10T01:51:28.828 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-10T01:51:28.828 INFO:teuthology.orchestra.run.smithi007.stdout:kind: ClusterRoleBinding 2022-03-10T01:51:28.829 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-10T01:51:28.829 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-10T01:51:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-10T01:51:28.829 INFO:teuthology.orchestra.run.smithi007.stdout:roleRef: 2022-03-10T01:51:28.829 INFO:teuthology.orchestra.run.smithi007.stdout: apiGroup: rbac.authorization.k8s.io 2022-03-10T01:51:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: kind: ClusterRole 2022-03-10T01:51:28.830 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-10T01:51:28.830 INFO:teuthology.orchestra.run.smithi007.stdout:subjects: 2022-03-10T01:51:28.830 INFO:teuthology.orchestra.run.smithi007.stdout:- kind: ServiceAccount 2022-03-10T01:51:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-10T01:51:28.831 INFO:teuthology.orchestra.run.smithi007.stdout: namespace: kube-system 2022-03-10T01:51:28.831 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-10T01:51:28.831 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: v1 2022-03-10T01:51:28.832 INFO:teuthology.orchestra.run.smithi007.stdout:kind: ServiceAccount 2022-03-10T01:51:28.832 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-10T01:51:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: name: flannel 2022-03-10T01:51:28.832 INFO:teuthology.orchestra.run.smithi007.stdout: namespace: kube-system 2022-03-10T01:51:28.832 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-10T01:51:28.833 INFO:teuthology.orchestra.run.smithi007.stdout:kind: ConfigMap 2022-03-10T01:51:28.834 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: v1 2022-03-10T01:51:28.834 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-10T01:51:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: name: kube-flannel-cfg 2022-03-10T01:51:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: namespace: kube-system 2022-03-10T01:51:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: labels: 2022-03-10T01:51:28.835 INFO:teuthology.orchestra.run.smithi007.stdout: tier: node 2022-03-10T01:51:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: app: flannel 2022-03-10T01:51:28.836 INFO:teuthology.orchestra.run.smithi007.stdout:data: 2022-03-10T01:51:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: cni-conf.json: | 2022-03-10T01:51:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: { 2022-03-10T01:51:28.836 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cbr0", 2022-03-10T01:51:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "cniVersion": "0.3.1", 2022-03-10T01:51:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: "plugins": [ 2022-03-10T01:51:28.837 INFO:teuthology.orchestra.run.smithi007.stdout: { 2022-03-10T01:51:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "flannel", 2022-03-10T01:51:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "delegate": { 2022-03-10T01:51:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "hairpinMode": true, 2022-03-10T01:51:28.838 INFO:teuthology.orchestra.run.smithi007.stdout: "isDefaultGateway": true 2022-03-10T01:51:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-10T01:51:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2022-03-10T01:51:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: { 2022-03-10T01:51:28.839 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "portmap", 2022-03-10T01:51:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "capabilities": { 2022-03-10T01:51:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: "portMappings": true 2022-03-10T01:51:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-10T01:51:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-10T01:51:28.840 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2022-03-10T01:51:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-10T01:51:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: net-conf.json: | 2022-03-10T01:51:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: { 2022-03-10T01:51:28.841 INFO:teuthology.orchestra.run.smithi007.stdout: "Network": "10.244.0.0/16", 2022-03-10T01:51:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: "Backend": { 2022-03-10T01:51:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: "Type": "vxlan" 2022-03-10T01:51:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-10T01:51:28.842 INFO:teuthology.orchestra.run.smithi007.stdout: } 2022-03-10T01:51:28.842 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2022-03-10T01:51:28.843 INFO:teuthology.orchestra.run.smithi007.stdout:apiVersion: apps/v1 2022-03-10T01:51:28.844 INFO:teuthology.orchestra.run.smithi007.stdout:kind: DaemonSet 2022-03-10T01:51:28.844 INFO:teuthology.orchestra.run.smithi007.stdout:metadata: 2022-03-10T01:51:28.844 INFO:teuthology.orchestra.run.smithi007.stdout: name: kube-flannel-ds 2022-03-10T01:51:28.845 INFO:teuthology.orchestra.run.smithi007.stdout: namespace: kube-system 2022-03-10T01:51:28.845 INFO:teuthology.orchestra.run.smithi007.stdout: labels: 2022-03-10T01:51:28.845 INFO:teuthology.orchestra.run.smithi007.stdout: tier: node 2022-03-10T01:51:28.845 INFO:teuthology.orchestra.run.smithi007.stdout: app: flannel 2022-03-10T01:51:28.846 INFO:teuthology.orchestra.run.smithi007.stdout:spec: 2022-03-10T01:51:28.846 INFO:teuthology.orchestra.run.smithi007.stdout: selector: 2022-03-10T01:51:28.846 INFO:teuthology.orchestra.run.smithi007.stdout: matchLabels: 2022-03-10T01:51:28.846 INFO:teuthology.orchestra.run.smithi007.stdout: app: flannel 2022-03-10T01:51:28.847 INFO:teuthology.orchestra.run.smithi007.stdout: template: 2022-03-10T01:51:28.847 INFO:teuthology.orchestra.run.smithi007.stdout: metadata: 2022-03-10T01:51:28.847 INFO:teuthology.orchestra.run.smithi007.stdout: labels: 2022-03-10T01:51:28.847 INFO:teuthology.orchestra.run.smithi007.stdout: tier: node 2022-03-10T01:51:28.847 INFO:teuthology.orchestra.run.smithi007.stdout: app: flannel 2022-03-10T01:51:28.848 INFO:teuthology.orchestra.run.smithi007.stdout: spec: 2022-03-10T01:51:28.848 INFO:teuthology.orchestra.run.smithi007.stdout: affinity: 2022-03-10T01:51:28.848 INFO:teuthology.orchestra.run.smithi007.stdout: nodeAffinity: 2022-03-10T01:51:28.848 INFO:teuthology.orchestra.run.smithi007.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-03-10T01:51:28.849 INFO:teuthology.orchestra.run.smithi007.stdout: nodeSelectorTerms: 2022-03-10T01:51:28.849 INFO:teuthology.orchestra.run.smithi007.stdout: - matchExpressions: 2022-03-10T01:51:28.849 INFO:teuthology.orchestra.run.smithi007.stdout: - key: kubernetes.io/os 2022-03-10T01:51:28.849 INFO:teuthology.orchestra.run.smithi007.stdout: operator: In 2022-03-10T01:51:28.849 INFO:teuthology.orchestra.run.smithi007.stdout: values: 2022-03-10T01:51:28.850 INFO:teuthology.orchestra.run.smithi007.stdout: - linux 2022-03-10T01:51:28.850 INFO:teuthology.orchestra.run.smithi007.stdout: hostNetwork: true 2022-03-10T01:51:28.850 INFO:teuthology.orchestra.run.smithi007.stdout: priorityClassName: system-node-critical 2022-03-10T01:51:28.850 INFO:teuthology.orchestra.run.smithi007.stdout: tolerations: 2022-03-10T01:51:28.851 INFO:teuthology.orchestra.run.smithi007.stdout: - operator: Exists 2022-03-10T01:51:28.851 INFO:teuthology.orchestra.run.smithi007.stdout: effect: NoSchedule 2022-03-10T01:51:28.851 INFO:teuthology.orchestra.run.smithi007.stdout: serviceAccountName: flannel 2022-03-10T01:51:28.851 INFO:teuthology.orchestra.run.smithi007.stdout: initContainers: 2022-03-10T01:51:28.852 INFO:teuthology.orchestra.run.smithi007.stdout: - name: install-cni-plugin 2022-03-10T01:51:28.852 INFO:teuthology.orchestra.run.smithi007.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-10T01:51:28.852 INFO:teuthology.orchestra.run.smithi007.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-03-10T01:51:28.852 INFO:teuthology.orchestra.run.smithi007.stdout: command: 2022-03-10T01:51:28.852 INFO:teuthology.orchestra.run.smithi007.stdout: - cp 2022-03-10T01:51:28.853 INFO:teuthology.orchestra.run.smithi007.stdout: args: 2022-03-10T01:51:28.853 INFO:teuthology.orchestra.run.smithi007.stdout: - -f 2022-03-10T01:51:28.853 INFO:teuthology.orchestra.run.smithi007.stdout: - /flannel 2022-03-10T01:51:28.853 INFO:teuthology.orchestra.run.smithi007.stdout: - /opt/cni/bin/flannel 2022-03-10T01:51:28.854 INFO:teuthology.orchestra.run.smithi007.stdout: volumeMounts: 2022-03-10T01:51:28.854 INFO:teuthology.orchestra.run.smithi007.stdout: - name: cni-plugin 2022-03-10T01:51:28.854 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /opt/cni/bin 2022-03-10T01:51:28.854 INFO:teuthology.orchestra.run.smithi007.stdout: - name: install-cni 2022-03-10T01:51:28.855 INFO:teuthology.orchestra.run.smithi007.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-10T01:51:28.855 INFO:teuthology.orchestra.run.smithi007.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-10T01:51:28.855 INFO:teuthology.orchestra.run.smithi007.stdout: command: 2022-03-10T01:51:28.855 INFO:teuthology.orchestra.run.smithi007.stdout: - cp 2022-03-10T01:51:28.856 INFO:teuthology.orchestra.run.smithi007.stdout: args: 2022-03-10T01:51:28.856 INFO:teuthology.orchestra.run.smithi007.stdout: - -f 2022-03-10T01:51:28.856 INFO:teuthology.orchestra.run.smithi007.stdout: - /etc/kube-flannel/cni-conf.json 2022-03-10T01:51:28.857 INFO:teuthology.orchestra.run.smithi007.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-03-10T01:51:28.857 INFO:teuthology.orchestra.run.smithi007.stdout: volumeMounts: 2022-03-10T01:51:28.857 INFO:teuthology.orchestra.run.smithi007.stdout: - name: cni 2022-03-10T01:51:28.858 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /etc/cni/net.d 2022-03-10T01:51:28.858 INFO:teuthology.orchestra.run.smithi007.stdout: - name: flannel-cfg 2022-03-10T01:51:28.858 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /etc/kube-flannel/ 2022-03-10T01:51:28.859 INFO:teuthology.orchestra.run.smithi007.stdout: containers: 2022-03-10T01:51:28.859 INFO:teuthology.orchestra.run.smithi007.stdout: - name: kube-flannel 2022-03-10T01:51:28.859 INFO:teuthology.orchestra.run.smithi007.stdout: #image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-10T01:51:28.859 INFO:teuthology.orchestra.run.smithi007.stdout: image: rancher/mirrored-flannelcni-flannel:v0.17.0 2022-03-10T01:51:28.859 INFO:teuthology.orchestra.run.smithi007.stdout: command: 2022-03-10T01:51:28.860 INFO:teuthology.orchestra.run.smithi007.stdout: - /opt/bin/flanneld 2022-03-10T01:51:28.860 INFO:teuthology.orchestra.run.smithi007.stdout: args: 2022-03-10T01:51:28.860 INFO:teuthology.orchestra.run.smithi007.stdout: - --ip-masq 2022-03-10T01:51:28.860 INFO:teuthology.orchestra.run.smithi007.stdout: - --kube-subnet-mgr 2022-03-10T01:51:28.861 INFO:teuthology.orchestra.run.smithi007.stdout: resources: 2022-03-10T01:51:28.861 INFO:teuthology.orchestra.run.smithi007.stdout: requests: 2022-03-10T01:51:28.861 INFO:teuthology.orchestra.run.smithi007.stdout: cpu: "100m" 2022-03-10T01:51:28.861 INFO:teuthology.orchestra.run.smithi007.stdout: memory: "50Mi" 2022-03-10T01:51:28.861 INFO:teuthology.orchestra.run.smithi007.stdout: limits: 2022-03-10T01:51:28.862 INFO:teuthology.orchestra.run.smithi007.stdout: cpu: "100m" 2022-03-10T01:51:28.862 INFO:teuthology.orchestra.run.smithi007.stdout: memory: "50Mi" 2022-03-10T01:51:28.862 INFO:teuthology.orchestra.run.smithi007.stdout: securityContext: 2022-03-10T01:51:28.862 INFO:teuthology.orchestra.run.smithi007.stdout: privileged: false 2022-03-10T01:51:28.863 INFO:teuthology.orchestra.run.smithi007.stdout: capabilities: 2022-03-10T01:51:28.863 INFO:teuthology.orchestra.run.smithi007.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-03-10T01:51:28.863 INFO:teuthology.orchestra.run.smithi007.stdout: env: 2022-03-10T01:51:28.863 INFO:teuthology.orchestra.run.smithi007.stdout: - name: POD_NAME 2022-03-10T01:51:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: valueFrom: 2022-03-10T01:51:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: fieldRef: 2022-03-10T01:51:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: fieldPath: metadata.name 2022-03-10T01:51:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: - name: POD_NAMESPACE 2022-03-10T01:51:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: valueFrom: 2022-03-10T01:51:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: fieldRef: 2022-03-10T01:51:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: fieldPath: metadata.namespace 2022-03-10T01:51:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: volumeMounts: 2022-03-10T01:51:28.865 INFO:teuthology.orchestra.run.smithi007.stdout: - name: run 2022-03-10T01:51:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /run/flannel 2022-03-10T01:51:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: - name: flannel-cfg 2022-03-10T01:51:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /etc/kube-flannel/ 2022-03-10T01:51:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: - name: xtables-lock 2022-03-10T01:51:28.866 INFO:teuthology.orchestra.run.smithi007.stdout: mountPath: /run/xtables.lock 2022-03-10T01:51:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: volumes: 2022-03-10T01:51:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: - name: run 2022-03-10T01:51:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: hostPath: 2022-03-10T01:51:28.867 INFO:teuthology.orchestra.run.smithi007.stdout: path: /run/flannel 2022-03-10T01:51:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: - name: cni-plugin 2022-03-10T01:51:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: hostPath: 2022-03-10T01:51:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: path: /opt/cni/bin 2022-03-10T01:51:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: - name: cni 2022-03-10T01:51:28.868 INFO:teuthology.orchestra.run.smithi007.stdout: hostPath: 2022-03-10T01:51:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: path: /etc/cni/net.d 2022-03-10T01:51:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: - name: flannel-cfg 2022-03-10T01:51:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: configMap: 2022-03-10T01:51:28.869 INFO:teuthology.orchestra.run.smithi007.stdout: name: kube-flannel-cfg 2022-03-10T01:51:28.870 INFO:teuthology.orchestra.run.smithi007.stdout: - name: xtables-lock 2022-03-10T01:51:28.870 INFO:teuthology.orchestra.run.smithi007.stdout: hostPath: 2022-03-10T01:51:28.870 INFO:teuthology.orchestra.run.smithi007.stdout: path: /run/xtables.lock 2022-03-10T01:51:28.870 INFO:teuthology.orchestra.run.smithi007.stdout: type: FileOrCreate 2022-03-10T01:51:28.872 INFO:teuthology.orchestra.run.smithi007.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 58680 0 --:--:-- --:--:-- --:--:-- 59291 2022-03-10T01:51:28.900 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-10T01:51:28.900 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.248.48.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-10T01:51:28.916 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.248.48.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.17.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-03-10T01:51:28.917 DEBUG:teuthology.orchestra.run.smithi007:> kubectl apply -f - 2022-03-10T01:51:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-10T01:51:31.282 INFO:teuthology.orchestra.run.smithi007.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-03-10T01:51:31.289 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-03-10T01:51:31.358 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-03-10T01:51:31.364 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/flannel created 2022-03-10T01:51:31.375 INFO:teuthology.orchestra.run.smithi007.stdout:configmap/kube-flannel-cfg created 2022-03-10T01:51:31.413 INFO:teuthology.orchestra.run.smithi007.stdout:daemonset.apps/kube-flannel-ds created 2022-03-10T01:51:31.418 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-10T01:51:31.418 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:31.418 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:31.425 INFO:tasks.kubeadm:Scratch devices on smithi007: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T01:51:31.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T01:51:31.447 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-10T01:51:31.447 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-10T01:51:31.447 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108072 s, 970 MB/s 2022-03-10T01:51:31.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T01:51:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-10T01:51:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-10T01:51:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119262 s, 879 MB/s 2022-03-10T01:51:31.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T01:51:31.548 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-10T01:51:31.548 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-10T01:51:31.548 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111956 s, 937 MB/s 2022-03-10T01:51:31.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T01:51:31.571 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-10T01:51:31.572 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-10T01:51:31.573 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0102787 s, 1.0 GB/s 2022-03-10T01:51:31.574 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:51:31.575 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:31.584 INFO:tasks.kubeadm:Scratch devices on smithi071: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T01:51:31.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T01:51:31.658 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records in 2022-03-10T01:51:31.658 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2022-03-10T01:51:31.658 INFO:teuthology.orchestra.run.smithi071.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0180416 s, 581 MB/s 2022-03-10T01:51:31.659 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T01:51:31.690 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records in 2022-03-10T01:51:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2022-03-10T01:51:31.691 INFO:teuthology.orchestra.run.smithi071.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.014989 s, 700 MB/s 2022-03-10T01:51:31.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T01:51:31.722 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records in 2022-03-10T01:51:31.723 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2022-03-10T01:51:31.723 INFO:teuthology.orchestra.run.smithi071.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156223 s, 671 MB/s 2022-03-10T01:51:31.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T01:51:31.799 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records in 2022-03-10T01:51:31.799 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2022-03-10T01:51:31.800 INFO:teuthology.orchestra.run.smithi071.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0161354 s, 650 MB/s 2022-03-10T01:51:31.801 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:51:31.802 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:31.811 INFO:tasks.kubeadm:Scratch devices on smithi184: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T01:51:31.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-10T01:51:31.885 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records in 2022-03-10T01:51:31.886 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2022-03-10T01:51:31.886 INFO:teuthology.orchestra.run.smithi184.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0167213 s, 627 MB/s 2022-03-10T01:51:31.887 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-10T01:51:31.914 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records in 2022-03-10T01:51:31.915 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2022-03-10T01:51:31.915 INFO:teuthology.orchestra.run.smithi184.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148902 s, 704 MB/s 2022-03-10T01:51:31.917 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-10T01:51:31.949 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records in 2022-03-10T01:51:31.950 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2022-03-10T01:51:31.950 INFO:teuthology.orchestra.run.smithi184.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0175695 s, 597 MB/s 2022-03-10T01:51:31.951 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-10T01:51:32.026 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records in 2022-03-10T01:51:32.027 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2022-03-10T01:51:32.027 INFO:teuthology.orchestra.run.smithi184.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0179174 s, 585 MB/s 2022-03-10T01:51:32.047 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-10T01:51:32.048 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi071-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi071 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi071-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi071 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi071-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi071 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi071-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi071 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi184-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi184 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi184-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi184 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi184-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi184 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi184-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi184 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-10T01:51:32.048 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f - 2022-03-10T01:51:32.502 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-1 created 2022-03-10T01:51:32.662 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-2 created 2022-03-10T01:51:32.668 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-3 created 2022-03-10T01:51:32.754 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-4 created 2022-03-10T01:51:32.822 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi071-lv-1 created 2022-03-10T01:51:32.846 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi071-lv-2 created 2022-03-10T01:51:32.854 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi071-lv-3 created 2022-03-10T01:51:32.915 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi071-lv-4 created 2022-03-10T01:51:32.998 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi184-lv-1 created 2022-03-10T01:51:33.002 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi184-lv-2 created 2022-03-10T01:51:33.025 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi184-lv-3 created 2022-03-10T01:51:33.039 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi184-lv-4 created 2022-03-10T01:51:33.091 INFO:teuthology.orchestra.run.smithi007.stdout:storageclass.storage.k8s.io/scratch created 2022-03-10T01:51:33.096 DEBUG:teuthology.orchestra.run.smithi007:> kubectl taint node smithi007 node-role.kubernetes.io/master- || true 2022-03-10T01:51:33.198 INFO:teuthology.orchestra.run.smithi007.stdout:node/smithi007 untainted 2022-03-10T01:51:33.202 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-10T01:51:33.202 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-10T01:51:33.215 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-10T01:51:33.216 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:33.216 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:33.224 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T01:51:33.224 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T01:51:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T01:51:33.275 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:33.275 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 498 Links: 1 2022-03-10T01:51:33.276 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:33.276 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-10 01:51:31.458473730 +0000 2022-03-10T01:51:33.276 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-10 01:51:31.458473730 +0000 2022-03-10T01:51:33.276 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-10 01:51:31.458473730 +0000 2022-03-10T01:51:33.276 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-10T01:51:33.284 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-10T01:51:33.285 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-10T01:51:33.285 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000339096 s, 1.5 MB/s 2022-03-10T01:51:33.289 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T01:51:33.295 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T01:51:33.296 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:33.296 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 503 Links: 1 2022-03-10T01:51:33.297 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:33.297 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-10 01:51:31.534473728 +0000 2022-03-10T01:51:33.298 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-10 01:51:31.534473728 +0000 2022-03-10T01:51:33.298 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-10 01:51:31.534473728 +0000 2022-03-10T01:51:33.298 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-10T01:51:33.300 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-10T01:51:33.301 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-10T01:51:33.301 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000179238 s, 2.9 MB/s 2022-03-10T01:51:33.306 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T01:51:33.311 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T01:51:33.311 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:33.312 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-03-10T01:51:33.312 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:33.312 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-10 01:51:31.558473727 +0000 2022-03-10T01:51:33.313 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-10 01:51:31.558473727 +0000 2022-03-10T01:51:33.313 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-10 01:51:31.558473727 +0000 2022-03-10T01:51:33.313 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-10T01:51:33.319 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-10T01:51:33.319 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-10T01:51:33.319 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000314784 s, 1.6 MB/s 2022-03-10T01:51:33.326 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T01:51:33.375 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T01:51:33.376 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:33.376 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-10T01:51:33.376 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:33.377 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-10 01:51:31.582473727 +0000 2022-03-10T01:51:33.377 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-10 01:51:31.582473727 +0000 2022-03-10T01:51:33.377 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-10 01:51:31.582473727 +0000 2022-03-10T01:51:33.377 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-10T01:51:33.385 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-10T01:51:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-10T01:51:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000263277 s, 1.9 MB/s 2022-03-10T01:51:33.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T01:51:33.495 INFO:teuthology.orchestra.run.smithi007.stdout:loop 2022-03-10T01:51:33.496 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_1... 2022-03-10T01:51:33.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T01:51:33.517 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-10T01:51:33.534 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_11 2022-03-10T01:51:33.553 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_2... 2022-03-10T01:51:33.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T01:51:33.624 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-10T01:51:33.649 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_21 2022-03-10T01:51:33.672 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_3... 2022-03-10T01:51:33.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T01:51:33.693 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-10T01:51:33.719 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_31 2022-03-10T01:51:33.742 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_4... 2022-03-10T01:51:33.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T01:51:33.763 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-10T01:51:33.784 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_41 2022-03-10T01:51:33.807 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:33.807 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:33.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo nvme list 2022-03-10T01:51:33.869 INFO:teuthology.orchestra.run.smithi007.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T01:51:33.870 INFO:teuthology.orchestra.run.smithi007.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T01:51:33.870 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme0n1 PHMB751300GE480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-10T01:51:33.870 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme1n1 4804ea92f5214c9e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:33.870 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme2n1 e7ce22f322f4e3fb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:33.871 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme3n1 04adb22e75a9dfc5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:33.871 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme4n1 59f96860b7d64bde Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:33.872 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T01:51:33.872 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:33.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/scratch_devs 2022-03-10T01:51:33.886 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:51:33.887 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:33.895 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T01:51:33.896 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T01:51:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T01:51:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:33.903 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-10T01:51:33.903 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:33.904 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2022-03-10 01:51:31.672696032 +0000 2022-03-10T01:51:33.904 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2022-03-10 01:51:31.672696032 +0000 2022-03-10T01:51:33.905 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2022-03-10 01:51:31.672696032 +0000 2022-03-10T01:51:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2022-03-10T01:51:33.912 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2022-03-10T01:51:33.913 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2022-03-10T01:51:33.913 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000576432 s, 888 kB/s 2022-03-10T01:51:33.917 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T01:51:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T01:51:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:33.963 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-10T01:51:33.963 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:33.964 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2022-03-10 01:51:31.700696032 +0000 2022-03-10T01:51:33.964 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2022-03-10 01:51:31.700696032 +0000 2022-03-10T01:51:33.964 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2022-03-10 01:51:31.700696032 +0000 2022-03-10T01:51:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2022-03-10T01:51:33.972 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2022-03-10T01:51:33.972 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2022-03-10T01:51:33.972 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000512257 s, 999 kB/s 2022-03-10T01:51:33.974 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T01:51:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T01:51:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:33.981 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-10T01:51:33.981 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:33.981 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2022-03-10 01:51:31.736696031 +0000 2022-03-10T01:51:33.981 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2022-03-10 01:51:31.736696031 +0000 2022-03-10T01:51:33.982 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2022-03-10 01:51:31.736696031 +0000 2022-03-10T01:51:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2022-03-10T01:51:33.988 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2022-03-10T01:51:33.988 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2022-03-10T01:51:33.988 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000417792 s, 1.2 MB/s 2022-03-10T01:51:33.992 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T01:51:33.997 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T01:51:33.998 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:33.998 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-10T01:51:33.998 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:33.999 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2022-03-10 01:51:31.816696029 +0000 2022-03-10T01:51:33.999 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2022-03-10 01:51:31.812696029 +0000 2022-03-10T01:51:33.999 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2022-03-10 01:51:31.812696029 +0000 2022-03-10T01:51:33.999 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2022-03-10T01:51:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2022-03-10T01:51:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2022-03-10T01:51:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000490791 s, 1.0 MB/s 2022-03-10T01:51:34.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T01:51:34.394 INFO:teuthology.orchestra.run.smithi071.stdout:loop 2022-03-10T01:51:34.395 INFO:tasks.nvme_loop:Connecting nvme_loop smithi071:/dev/vg_nvme/lv_1... 2022-03-10T01:51:34.395 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T01:51:34.416 INFO:teuthology.orchestra.run.smithi071.stdout:1 2022-03-10T01:51:34.439 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/vg_nvme/lv_11 2022-03-10T01:51:34.459 INFO:tasks.nvme_loop:Connecting nvme_loop smithi071:/dev/vg_nvme/lv_2... 2022-03-10T01:51:34.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T01:51:34.481 INFO:teuthology.orchestra.run.smithi071.stdout:1 2022-03-10T01:51:34.505 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/vg_nvme/lv_21 2022-03-10T01:51:34.523 INFO:tasks.nvme_loop:Connecting nvme_loop smithi071:/dev/vg_nvme/lv_3... 2022-03-10T01:51:34.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T01:51:34.544 INFO:teuthology.orchestra.run.smithi071.stdout:1 2022-03-10T01:51:34.566 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/vg_nvme/lv_31 2022-03-10T01:51:34.584 INFO:tasks.nvme_loop:Connecting nvme_loop smithi071:/dev/vg_nvme/lv_4... 2022-03-10T01:51:34.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T01:51:34.601 INFO:teuthology.orchestra.run.smithi071.stdout:1 2022-03-10T01:51:34.625 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/vg_nvme/lv_41 2022-03-10T01:51:34.639 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:51:34.640 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:34.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo nvme list 2022-03-10T01:51:34.704 INFO:teuthology.orchestra.run.smithi071.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T01:51:34.705 INFO:teuthology.orchestra.run.smithi071.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T01:51:34.705 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme0n1 PHFT62040172400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T01:51:34.705 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme1n1 13c06dbe6249cff6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:34.706 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme2n1 9a882199e427ceab Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:34.706 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme3n1 20fb562ff0d93de9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:34.706 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme4n1 bcf97c91c33b47be Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:34.706 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T01:51:34.707 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:51:34.707 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/scratch_devs 2022-03-10T01:51:34.722 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:51:34.722 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:34.728 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-10T01:51:34.729 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-10T01:51:34.775 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-10T01:51:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:34.776 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-10T01:51:34.776 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:34.777 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2022-03-10 01:51:31.904676583 +0000 2022-03-10T01:51:34.777 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2022-03-10 01:51:31.904676583 +0000 2022-03-10T01:51:34.777 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2022-03-10 01:51:31.904676583 +0000 2022-03-10T01:51:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2022-03-10T01:51:34.782 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2022-03-10T01:51:34.782 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2022-03-10T01:51:34.782 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000457604 s, 1.1 MB/s 2022-03-10T01:51:34.785 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-10T01:51:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-10T01:51:34.792 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:34.792 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-03-10T01:51:34.793 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:34.793 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2022-03-10 01:51:31.936676582 +0000 2022-03-10T01:51:34.793 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2022-03-10 01:51:31.936676582 +0000 2022-03-10T01:51:34.793 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2022-03-10 01:51:31.936676582 +0000 2022-03-10T01:51:34.793 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2022-03-10T01:51:34.799 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2022-03-10T01:51:34.799 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2022-03-10T01:51:34.799 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000470141 s, 1.1 MB/s 2022-03-10T01:51:34.802 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-10T01:51:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-10T01:51:34.808 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:34.809 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-10T01:51:34.809 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:34.809 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2022-03-10 01:51:31.972676581 +0000 2022-03-10T01:51:34.810 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2022-03-10 01:51:31.972676581 +0000 2022-03-10T01:51:34.810 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2022-03-10 01:51:31.972676581 +0000 2022-03-10T01:51:34.810 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2022-03-10T01:51:34.816 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2022-03-10T01:51:34.816 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2022-03-10T01:51:34.816 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000503822 s, 1.0 MB/s 2022-03-10T01:51:34.820 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-10T01:51:34.869 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-10T01:51:34.869 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-10T01:51:34.869 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-10T01:51:34.870 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-10T01:51:34.870 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2022-03-10 01:51:32.048676580 +0000 2022-03-10T01:51:34.870 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2022-03-10 01:51:32.044676580 +0000 2022-03-10T01:51:34.870 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2022-03-10 01:51:32.044676580 +0000 2022-03-10T01:51:34.871 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2022-03-10T01:51:34.879 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2022-03-10T01:51:34.880 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2022-03-10T01:51:34.880 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000555767 s, 921 kB/s 2022-03-10T01:51:34.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-10T01:51:35.049 INFO:teuthology.orchestra.run.smithi184.stdout:loop 2022-03-10T01:51:35.050 INFO:tasks.nvme_loop:Connecting nvme_loop smithi184:/dev/vg_nvme/lv_1... 2022-03-10T01:51:35.051 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-10T01:51:35.072 INFO:teuthology.orchestra.run.smithi184.stdout:1 2022-03-10T01:51:35.098 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_11 2022-03-10T01:51:35.114 INFO:tasks.nvme_loop:Connecting nvme_loop smithi184:/dev/vg_nvme/lv_2... 2022-03-10T01:51:35.115 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-10T01:51:35.134 INFO:teuthology.orchestra.run.smithi184.stdout:1 2022-03-10T01:51:35.157 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_21 2022-03-10T01:51:35.174 INFO:tasks.nvme_loop:Connecting nvme_loop smithi184:/dev/vg_nvme/lv_3... 2022-03-10T01:51:35.175 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-10T01:51:35.242 INFO:teuthology.orchestra.run.smithi184.stdout:1 2022-03-10T01:51:35.266 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_31 2022-03-10T01:51:35.287 INFO:tasks.nvme_loop:Connecting nvme_loop smithi184:/dev/vg_nvme/lv_4... 2022-03-10T01:51:35.287 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-10T01:51:35.306 INFO:teuthology.orchestra.run.smithi184.stdout:1 2022-03-10T01:51:35.328 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_41 2022-03-10T01:51:35.349 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:51:35.350 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:51:35.357 DEBUG:teuthology.orchestra.run.smithi184:> sudo nvme list 2022-03-10T01:51:35.411 INFO:teuthology.orchestra.run.smithi184.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-10T01:51:35.412 INFO:teuthology.orchestra.run.smithi184.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-10T01:51:35.412 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme0n1 CVFT626500CE400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-10T01:51:35.412 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme1n1 48d40998d8611949 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:35.413 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme2n1 ac71e51bacd0a4cb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:35.413 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme3n1 b9ed302b3f27519d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:35.413 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme4n1 aac02a5741c2e7cf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-10T01:51:35.415 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-10T01:51:35.415 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:51:35.416 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/scratch_devs 2022-03-10T01:51:35.425 INFO:teuthology.run_tasks:Running task rook... 2022-03-10T01:51:35.496 INFO:tasks.rook:Rook start 2022-03-10T01:51:35.497 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7a44e9ebca556862e9704be295a07719d7db47b3', 'rook_image': 'rook/ceph:master'} 2022-03-10T01:51:35.497 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 2022-03-10T01:51:35.498 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-10T01:51:35.498 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-10T01:51:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:Cloning into 'rook'... 2022-03-10T01:51:37.855 INFO:tasks.rook:/home/teuthworker 2022-03-10T01:51:37.856 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/sbin:/usr/sbin ip addr show 2022-03-10T01:51:37.861 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-10T01:51:37.862 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-10T01:51:37.862 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2022-03-10T01:51:37.862 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-10T01:51:37.863 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2022-03-10T01:51:37.863 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-10T01:51:37.863 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T01:51:37.863 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2022-03-10T01:51:37.864 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2022-03-10T01:51:37.864 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T01:51:37.864 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2022-03-10T01:51:37.864 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2022-03-10T01:51:37.864 INFO:teuthology.orchestra.run.smithi007.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-10T01:51:37.865 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2022-03-10T01:51:37.865 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2022-03-10T01:51:37.865 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-10T01:51:37.865 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42846sec preferred_lft 42846sec 2022-03-10T01:51:37.866 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2022-03-10T01:51:37.866 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-10T01:51:37.866 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-10T01:51:37.866 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2022-03-10T01:51:37.867 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2022-03-10T01:51:37.867 INFO:teuthology.orchestra.run.smithi007.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-10T01:51:37.867 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 02:42:58:cf:59:65 brd ff:ff:ff:ff:ff:ff 2022-03-10T01:51:37.867 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-10T01:51:37.867 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-10T01:51:37.869 DEBUG:teuthology.orchestra.run.smithi007:> which systemctl 2022-03-10T01:51:37.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/systemctl 2022-03-10T01:51:37.875 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-10T01:51:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-10T01:51:37.924 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-10T01:51:37.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo lsb_release -is 2022-03-10T01:51:37.995 INFO:teuthology.orchestra.run.smithi007.stdout:Ubuntu 2022-03-10T01:51:37.999 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-10T01:51:38.000 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-10T01:51:38.056 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T01:51:38.072 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:38.073 DEBUG:teuthology.orchestra.run.smithi007:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-10T01:51:38.080 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-10T01:51:38.126 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:51:38.127 DEBUG:teuthology.orchestra.run.smithi007:> dd of=operator.yaml 2022-03-10T01:51:38.134 INFO:tasks.rook:Deploying operator 2022-03-10T01:51:38.150 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T01:51:38.175 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T01:51:38.232 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-10T01:51:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-10T01:51:38.544 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-10T01:51:38.656 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-10T01:51:38.668 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-10T01:51:38.842 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-10T01:51:38.886 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-10T01:51:38.993 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-10T01:51:39.011 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-10T01:51:39.049 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-10T01:51:39.078 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-10T01:51:39.186 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-10T01:51:39.255 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-10T01:51:39.372 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-10T01:51:39.397 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-10T01:51:39.444 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-10T01:51:39.556 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-10T01:51:39.574 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-10T01:51:39.734 INFO:teuthology.orchestra.run.smithi007.stdout:namespace/rook-ceph created 2022-03-10T01:51:39.750 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-10T01:51:39.802 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-10T01:51:39.905 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-10T01:51:39.910 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T01:51:39.921 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-10T01:51:39.942 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-10T01:51:39.968 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-10T01:51:39.994 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-10T01:51:40.003 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-10T01:51:40.068 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-10T01:51:40.109 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T01:51:40.113 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T01:51:40.120 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-10T01:51:40.129 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-10T01:51:40.153 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T01:51:40.162 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-10T01:51:40.213 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-10T01:51:40.216 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-10T01:51:40.220 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-10T01:51:40.230 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T01:51:40.252 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T01:51:40.268 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-10T01:51:40.288 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-10T01:51:40.304 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-10T01:51:40.378 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-10T01:51:40.382 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-10T01:51:40.436 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-10T01:51:40.437 INFO:teuthology.orchestra.run.smithi007.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-10T01:51:40.461 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-10T01:51:40.470 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-10T01:51:40.521 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-10T01:51:40.523 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-10T01:51:40.546 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-10T01:51:40.570 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T01:51:40.579 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-10T01:51:40.588 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T01:51:40.596 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-10T01:51:40.603 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-10T01:51:40.671 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-10T01:51:40.679 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-10T01:51:40.704 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-10T01:51:40.720 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-10T01:51:40.746 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-10T01:51:40.762 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-10T01:51:40.839 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-10T01:51:40.865 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-10T01:51:40.889 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-10T01:51:40.905 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-10T01:51:40.979 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-10T01:51:40.983 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-10T01:51:40.987 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-10T01:51:41.015 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-mgr created 2022-03-10T01:51:41.020 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-osd created 2022-03-10T01:51:41.029 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-10T01:51:41.139 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-system created 2022-03-10T01:51:41.150 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-10T01:51:41.183 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-10T01:51:41.200 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-10T01:51:41.229 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-10T01:51:41.307 INFO:teuthology.orchestra.run.smithi007.stdout:configmap/rook-ceph-operator-config created 2022-03-10T01:51:41.327 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps/rook-ceph-operator created 2022-03-10T01:51:41.333 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:51:41.480 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:51:41.481 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-p2t8v 0/1 Pending 0 0s 2022-03-10T01:51:51.485 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:51:51.555 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:51:51.555 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-p2t8v 0/1 ContainerCreating 0 10s 2022-03-10T01:52:01.561 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:52:01.645 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:01.645 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-p2t8v 0/1 ContainerCreating 0 20s 2022-03-10T01:52:11.648 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:52:11.717 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:11.717 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-p2t8v 0/1 ContainerCreating 0 30s 2022-03-10T01:52:21.720 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-10T01:52:21.792 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:21.793 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-p2t8v 1/1 Running 0 40s 2022-03-10T01:52:21.795 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-p2t8v 2022-03-10T01:52:21.803 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:52:21.803 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:52:21.808 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:52:21.809 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:52:21.818 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:52:21.818 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2022-03-10T01:52:21.830 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-10T01:52:21.831 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-10T01:52:21.831 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-10T01:52:21.832 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-10T01:52:21.832 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-10T01:52:21.833 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-10T01:52:21.833 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-10T01:52:21.834 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-10T01:52:21.837 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f - 2022-03-10T01:52:21.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:18.900336 I | rookcmd: starting Rook v1.8.0-alpha.0.393.ga181988f6 with arguments '/usr/local/bin/rook ceph operator' 2022-03-10T01:52:21.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:18.900444 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-10T01:52:21.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:18.900448 I | cephcmd: starting Rook-Ceph operator 2022-03-10T01:52:21.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.088874 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-10T01:52:21.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.099168 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T01:52:21.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.099192 I | operator: watching all namespaces for Ceph CRs 2022-03-10T01:52:21.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.099256 I | operator: setting up schemes 2022-03-10T01:52:21.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.101874 I | operator: setting up the controller-runtime manager 2022-03-10T01:52:21.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.308474 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T01:52:21.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311698 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T01:52:21.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311746 I | ceph-cluster-controller: successfully started 2022-03-10T01:52:21.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311821 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T01:52:21.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311848 I | ceph-crashcollector-controller: successfully started 2022-03-10T01:52:21.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311888 I | ceph-block-pool-controller: successfully started 2022-03-10T01:52:21.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311927 I | ceph-object-store-user-controller: successfully started 2022-03-10T01:52:21.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311955 I | ceph-object-realm-controller: successfully started 2022-03-10T01:52:21.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311974 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T01:52:21.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.311991 I | ceph-object-zone-controller: successfully started 2022-03-10T01:52:21.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312114 I | ceph-object-controller: successfully started 2022-03-10T01:52:21.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312154 I | ceph-file-controller: successfully started 2022-03-10T01:52:21.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312198 I | ceph-nfs-controller: successfully started 2022-03-10T01:52:21.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312235 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T01:52:21.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312264 I | ceph-client-controller: successfully started 2022-03-10T01:52:21.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312289 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T01:52:21.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312327 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T01:52:21.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312349 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T01:52:21.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312370 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T01:52:21.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312389 I | ceph-bucket-topic: successfully started 2022-03-10T01:52:21.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312406 I | ceph-bucket-notification: successfully started 2022-03-10T01:52:21.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312431 I | ceph-bucket-notification: successfully started 2022-03-10T01:52:21.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.312445 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T01:52:21.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.313885 I | operator: starting the controller-runtime manager 2022-03-10T01:52:21.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.416331 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T01:52:21.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.416368 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T01:52:21.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.416388 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T01:52:21.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.417480 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-10T01:52:21.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.417505 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-10T01:52:21.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.419107 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T01:52:21.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.419523 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-10T01:52:21.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.529014 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-10T01:52:21.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.529035 I | ceph-csi: CSI CephFS driver disabled 2022-03-10T01:52:21.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.529045 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-10T01:52:21.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.531426 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-10T01:52:21.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:19.538393 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-10T01:52:22.781 INFO:teuthology.orchestra.run.smithi007.stdout:configmap/rook-config-override created 2022-03-10T01:52:22.786 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-10T01:52:22.786 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:52:22.786 DEBUG:teuthology.orchestra.run.smithi007:> dd of=cluster.yaml 2022-03-10T01:52:22.793 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f cluster.yaml 2022-03-10T01:52:25.088 INFO:teuthology.orchestra.run.smithi007.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-10T01:52:25.094 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.111937 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-10T01:52:25.102 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T01:52:25.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.130417 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:52:25.121 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-10T01:52:25.130 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.147703 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-10T01:52:25.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.164032 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T01:52:25.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.217234 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:52:25.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234333 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T01:52:25.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234359 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T01:52:25.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234368 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T01:52:25.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234376 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T01:52:25.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234384 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T01:52:25.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234392 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T01:52:25.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234401 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T01:52:25.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234415 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T01:52:25.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234432 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T01:52:25.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234448 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T01:52:25.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234465 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T01:52:25.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234482 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T01:52:25.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234500 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T01:52:25.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234510 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T01:52:25.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234521 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T01:52:25.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234535 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T01:52:25.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234718 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:52:25.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:52:25.234741 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:52:25.432 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps/rook-ceph-tools created 2022-03-10T01:52:25.436 INFO:tasks.rook:Waiting for tools container to start 2022-03-10T01:52:25.437 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:52:25.561 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:25.561 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 Pending 0 0s 2022-03-10T01:52:30.565 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:52:30.644 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:30.644 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 5s 2022-03-10T01:52:35.647 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:52:35.716 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:35.717 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 10s 2022-03-10T01:52:40.719 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:52:40.790 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:40.790 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 15s 2022-03-10T01:52:45.794 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:52:45.864 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:45.864 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 20s 2022-03-10T01:52:50.866 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:52:50.930 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:50.930 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 25s 2022-03-10T01:52:55.932 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:52:56.001 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:52:56.001 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 31s 2022-03-10T01:53:01.005 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:01.068 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:01.068 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 36s 2022-03-10T01:53:06.072 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:06.132 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:06.133 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 41s 2022-03-10T01:53:11.135 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:11.201 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:11.202 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 46s 2022-03-10T01:53:16.204 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:16.283 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:16.284 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 51s 2022-03-10T01:53:21.287 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:21.359 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:21.359 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 56s 2022-03-10T01:53:26.362 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:26.434 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:26.434 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 61s 2022-03-10T01:53:31.437 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:31.505 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:31.505 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 66s 2022-03-10T01:53:36.508 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:36.580 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:36.580 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 71s 2022-03-10T01:53:41.583 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:41.654 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:41.655 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 76s 2022-03-10T01:53:46.657 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:46.732 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:46.733 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 81s 2022-03-10T01:53:51.736 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:51.802 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:51.803 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 86s 2022-03-10T01:53:56.805 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:53:56.880 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:53:56.880 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 91s 2022-03-10T01:54:00.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860496 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T01:54:00.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860534 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T01:54:00.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860546 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:54:00.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860556 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:54:00.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860567 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:54:00.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860577 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:54:00.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860584 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:54:00.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860591 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:54:00.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860597 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T01:54:00.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860604 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T01:54:00.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860611 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:54:00.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860618 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:54:00.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860624 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:54:00.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860635 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:54:00.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860646 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T01:54:00.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860657 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T01:54:00.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860664 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T01:54:00.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860671 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T01:54:00.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860684 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T01:54:00.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860691 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:54:00.857 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860698 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:54:00.857 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860705 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T01:54:00.857 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860716 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T01:54:00.857 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860728 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T01:54:00.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.860735 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T01:54:00.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.886532 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T01:54:00.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.899575 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:54:00.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.899603 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:54:00.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.899608 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:54:00.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.899612 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:54:00.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.899616 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:54:00.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.899620 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:54:00.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.899624 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T01:54:00.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.922134 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:54:00.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.922161 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:54:00.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:00.922168 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:54:01.034 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.047371 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T01:54:01.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.182864 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:54:01.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.182899 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:54:01.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.182912 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T01:54:01.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.195988 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:54:01.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.196022 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:54:01.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.196031 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:54:01.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.252580 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T01:54:01.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.379036 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:54:01.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.611881 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T01:54:01.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:01.612031 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:54:01.883 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:02.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:02.080529 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T01:54:02.366 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:02.366 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 97s 2022-03-10T01:54:07.369 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:07.432 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:07.433 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 102s 2022-03-10T01:54:12.436 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:12.510 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:12.510 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 107s 2022-03-10T01:54:13.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:13.968505 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:54:13.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:13.968532 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:54:13.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:13.968545 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:54:13.964 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:13.978355 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:54:14.016 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:14.029715 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-10 01:54:14.001792018 +0000 UTC m=+115.129291120 LastTransitionTime:2022-03-10 01:54:14.001791856 +0000 UTC m=+115.129290974}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:54:14.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:14.043873 I | op-mon: start running mons 2022-03-10T01:54:14.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:14.189856 I | op-mon: creating mon secrets for a new cluster 2022-03-10T01:54:14.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:14.209956 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-10T01:54:14.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:14.229102 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-10T01:54:14.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:14.886006 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:54:14.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:14.886311 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:54:16.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.093272 I | op-mon: parsing mon endpoints: 2022-03-10T01:54:16.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.093322 W | op-mon: ignoring invalid monitor 2022-03-10T01:54:16.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.093357 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T01:54:16.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.093367 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:54:16.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.095342 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T01:54:16.083 INFO:tasks.rook.operator.smithi007.stdout:I0310 01:54:16.095432 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:54:16.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.886436 I | op-mon: targeting the mon count 3 2022-03-10T01:54:16.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.893287 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-10T01:54:16.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.920508 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-10T01:54:16.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:16.946111 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-10T01:54:17.513 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:17.579 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:17.580 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 112s 2022-03-10T01:54:17.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:17.687748 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi071 2022-03-10T01:54:17.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:17.687782 I | op-mon: mon a assigned to node smithi071 2022-03-10T01:54:17.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:17.886827 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi184 2022-03-10T01:54:17.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:17.886859 I | op-mon: mon b assigned to node smithi184 2022-03-10T01:54:18.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:18.085815 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi007 2022-03-10T01:54:18.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:18.085844 I | op-mon: mon c assigned to node smithi007 2022-03-10T01:54:18.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:18.093889 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-10T01:54:18.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:18.100046 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-10T01:54:18.615 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:18.629378 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-10T01:54:18.945 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:18.952681 I | op-mon: creating mon a 2022-03-10T01:54:19.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:19.447716 I | op-mon: mon "a" endpoint is [v2:10.99.228.129:3300,v1:10.99.228.129:6789] 2022-03-10T01:54:19.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:19.872578 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-10T01:54:19.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:19.872627 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-10T01:54:19.862 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:19.873035 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789"]}] data:a=10.99.228.129:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:-1] 2022-03-10T01:54:20.094 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:20.108208 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:54:20.201 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:20.215216 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:54:20.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:20.242382 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-10T01:54:20.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:20.274493 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-10T01:54:20.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:20.887513 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789"]}] data:a=10.99.228.129:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:0] 2022-03-10T01:54:20.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:20.887545 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:54:21.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:21.089548 I | op-mon: mon a is not yet running 2022-03-10T01:54:21.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:21.089576 I | op-mon: mons running: [] 2022-03-10T01:54:22.581 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:22.648 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:22.649 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 117s 2022-03-10T01:54:27.652 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:27.725 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:27.726 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 2m2s 2022-03-10T01:54:32.728 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:32.798 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:32.799 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 2m7s 2022-03-10T01:54:37.801 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:37.874 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:37.875 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 2m12s 2022-03-10T01:54:41.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:41.283202 I | op-mon: mons running: [a] 2022-03-10T01:54:42.877 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:42.943 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:42.944 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 0/1 ContainerCreating 0 2m17s 2022-03-10T01:54:47.946 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-10T01:54:48.017 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-10T01:54:48.017 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-lc2pk 1/1 Running 0 2m23s 2022-03-10T01:54:48.019 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-10T01:54:48.020 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:54:52.569 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:54:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:54:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:54:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:54:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:54:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:54:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:54:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:54:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:54:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:54:52.575 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:54:52.575 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:54:52.575 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:54:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:54:52.577 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:54:52.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:52.773385 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:54:52.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:52.773413 I | op-mon: mons created: 1 2022-03-10T01:54:53.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:53.093864 I | op-mon: waiting for mon quorum with [a] 2022-03-10T01:54:53.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:53.436684 I | op-mon: mons running: [a] 2022-03-10T01:54:53.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:53.694549 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:54:53.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:53.694573 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:54:54.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:54.044876 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:54:54.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:54.044901 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:54:54.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:54.383015 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:54:54.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:54.383037 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:54:54.692 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:54.706077 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:54:54.692 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:54.706100 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:54:55.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.028361 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:54:55.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.028394 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:54:55.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.291960 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:54:55.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.291993 I | op-mon: creating mon b 2022-03-10T01:54:55.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.428057 I | op-mon: mon "a" endpoint is [v2:10.99.228.129:3300,v1:10.99.228.129:6789] 2022-03-10T01:54:55.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.458812 I | op-mon: mon "b" endpoint is [v2:10.101.240.174:3300,v1:10.101.240.174:6789] 2022-03-10T01:54:55.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.552074 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789","10.101.240.174:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:0] 2022-03-10T01:54:55.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.664686 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:54:55.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.668935 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:54:55.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.939724 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T01:54:55.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.953482 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T01:54:55.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:55.953514 I | op-mon: waiting for mon quorum with [a b] 2022-03-10T01:54:56.308 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:56.317618 I | op-mon: mons running: [a b] 2022-03-10T01:54:56.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:56.583126 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:54:56.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:56.679235 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-10T01:54:57.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:57.297511 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789","10.101.240.174:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:1] 2022-03-10T01:54:57.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:57.297532 I | op-mon: waiting for mon quorum with [a b] 2022-03-10T01:54:57.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:57.700011 I | op-mon: mons running: [a b] 2022-03-10T01:54:57.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:57.966099 I | op-mon: Monitors in quorum: [a] 2022-03-10T01:54:57.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:57.966118 I | op-mon: mons created: 2 2022-03-10T01:54:58.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:58.471518 I | op-mon: waiting for mon quorum with [a b] 2022-03-10T01:54:58.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:58.485045 I | op-mon: mons running: [a b] 2022-03-10T01:54:58.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:58.899391 W | op-mon: failed to find initial monitor b in mon map 2022-03-10T01:54:58.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:54:58.899408 W | op-mon: monitor b is not in quorum list 2022-03-10T01:55:02.578 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:55:03.495 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:55:03.495 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:55:03.496 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:55:03.496 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:55:03.496 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:55:03.497 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:55:03.497 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:55:03.497 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:55:03.497 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:55:03.498 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:55:03.498 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:55:03.498 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:55:03.498 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:55:03.499 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:55:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:55:04.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:04.042256 I | op-mon: mons running: [a b] 2022-03-10T01:55:04.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:04.252950 W | op-mon: failed to find initial monitor b in mon map 2022-03-10T01:55:04.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:04.252974 W | op-mon: monitor b is not in quorum list 2022-03-10T01:55:09.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:09.646637 I | op-mon: mon b is not yet running 2022-03-10T01:55:09.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:09.646669 I | op-mon: mons running: [a] 2022-03-10T01:55:13.502 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:55:14.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:14.711520 I | op-mon: mon b is not yet running 2022-03-10T01:55:14.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:14.711548 I | op-mon: mons running: [a] 2022-03-10T01:55:14.704 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:55:14.707 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:55:14.707 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:55:14.707 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:55:14.708 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:55:14.708 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:55:14.708 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:55:14.708 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:55:14.708 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:55:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:55:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:55:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:55:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:55:14.710 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:55:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:55:19.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:19.883753 I | op-mon: mon b is not yet running 2022-03-10T01:55:19.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:19.883779 I | op-mon: mons running: [a] 2022-03-10T01:55:24.714 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:55:24.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:24.908449 I | op-mon: mon b is not yet running 2022-03-10T01:55:24.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:24.908485 I | op-mon: mons running: [a] 2022-03-10T01:55:25.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:55:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:55:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:55:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:55:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:55:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:55:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:55:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:55:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:55:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:55:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:55:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:55:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:55:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:55:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:55:30.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:30.621749 I | op-mon: mon b is not yet running 2022-03-10T01:55:30.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:30.621775 I | op-mon: mons running: [a] 2022-03-10T01:55:35.344 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:55:35.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:35.750479 I | op-mon: mon b is not yet running 2022-03-10T01:55:35.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:35.750515 I | op-mon: mons running: [a] 2022-03-10T01:55:36.136 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:55:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:55:36.136 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:55:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:55:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:55:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:55:36.137 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:55:36.138 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:55:36.138 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:55:36.138 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:55:36.138 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:55:36.138 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:55:36.139 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:55:36.139 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:55:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:55:40.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:40.770623 I | op-mon: mon b is not yet running 2022-03-10T01:55:40.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:40.770647 I | op-mon: mons running: [a] 2022-03-10T01:55:45.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:45.850478 I | op-mon: mon b is not yet running 2022-03-10T01:55:45.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:45.850551 I | op-mon: mons running: [a] 2022-03-10T01:55:46.144 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:55:46.605 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:55:46.605 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:55:46.606 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:55:46.606 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:55:46.606 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:55:46.606 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:55:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:55:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:55:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:55:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:55:46.607 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:55:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:55:46.608 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:55:46.609 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:55:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:55:50.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:50.873497 I | op-mon: mons running: [a b] 2022-03-10T01:55:54.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:54.323182 I | op-mon: Monitors in quorum: [a b] 2022-03-10T01:55:54.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:54.323207 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:55:54.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:54.546901 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:55:54.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:54.546941 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:55:54.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:54.760498 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:55:54.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:54.760531 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:55:54.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:54.976040 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:55:54.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:54.976068 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:55:55.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:55.207154 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:55:55.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:55.207183 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:55:55.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:55.436286 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:55:55.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:55.436318 I | op-mon: creating mon c 2022-03-10T01:55:55.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:55.566945 I | op-mon: mon "a" endpoint is [v2:10.99.228.129:3300,v1:10.99.228.129:6789] 2022-03-10T01:55:55.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:55.724596 I | op-mon: mon "b" endpoint is [v2:10.101.240.174:3300,v1:10.101.240.174:6789] 2022-03-10T01:55:55.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:55.754456 I | op-mon: mon "c" endpoint is [v2:10.111.0.148:3300,v1:10.111.0.148:6789] 2022-03-10T01:55:55.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:55.854513 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789","10.101.240.174:6789","10.111.0.148:6789"]}] data:b=10.101.240.174:6789,c=10.111.0.148:6789,a=10.99.228.129:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:1] 2022-03-10T01:55:56.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:56.440926 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:55:56.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:56.441226 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:55:56.613 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:55:56.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:56.865860 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T01:55:56.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:56.877362 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T01:55:56.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:56.877398 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T01:55:57.121 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:55:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:55:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:55:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:55:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:55:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:55:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:55:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:55:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:55:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:55:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:55:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:55:57.125 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:55:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:55:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:55:57.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:57.445796 I | op-mon: mons running: [a b c] 2022-03-10T01:55:57.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:57.672228 I | op-mon: Monitors in quorum: [a b] 2022-03-10T01:55:57.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:57.680431 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T01:55:57.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:57.689624 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T01:55:57.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:57.689643 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T01:55:58.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:58.044738 I | op-mon: mons running: [a b c] 2022-03-10T01:55:58.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:58.281064 I | op-mon: Monitors in quorum: [a b] 2022-03-10T01:55:58.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:58.320786 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-10T01:55:59.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:59.276443 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.101.240.174:6789","10.111.0.148:6789","10.99.228.129:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-10T01:55:59.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:59.276483 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T01:55:59.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:59.669474 I | op-mon: mons running: [a b c] 2022-03-10T01:55:59.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:59.910307 I | op-mon: Monitors in quorum: [a b] 2022-03-10T01:55:59.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:55:59.910342 I | op-mon: mons created: 3 2022-03-10T01:56:00.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:00.153233 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T01:56:00.230 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:00.244131 I | op-mon: mons running: [a b c] 2022-03-10T01:56:00.453 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:00.467378 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T01:56:00.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:00.467414 W | op-mon: monitor c is not in quorum list 2022-03-10T01:56:05.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:05.495419 I | op-mon: mons running: [a b c] 2022-03-10T01:56:05.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:05.721653 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T01:56:05.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:05.721685 W | op-mon: monitor c is not in quorum list 2022-03-10T01:56:07.128 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:56:07.611 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:56:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:56:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:56:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:56:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:56:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:56:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:56:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:56:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:56:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:56:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:56:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:56:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:56:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:56:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:56:10.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:10.743970 I | op-mon: mons running: [a b c] 2022-03-10T01:56:10.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:10.995706 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T01:56:10.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:10.995739 W | op-mon: monitor c is not in quorum list 2022-03-10T01:56:16.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:16.014685 I | op-mon: mons running: [a b c] 2022-03-10T01:56:16.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:16.251431 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T01:56:16.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:16.251466 W | op-mon: monitor c is not in quorum list 2022-03-10T01:56:17.619 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:56:18.652 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:56:18.653 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:56:18.653 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:56:18.654 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:56:18.654 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:56:18.655 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:56:18.655 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:56:18.656 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:56:18.656 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:56:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:56:18.657 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:56:18.658 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:56:18.658 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:56:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:56:18.661 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:56:21.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:21.273396 I | op-mon: mons running: [a b c] 2022-03-10T01:56:21.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:21.515337 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T01:56:21.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:21.515363 W | op-mon: monitor c is not in quorum list 2022-03-10T01:56:27.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:27.214048 I | op-mon: mons running: [a b c] 2022-03-10T01:56:27.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:27.463753 W | op-mon: failed to find initial monitor c in mon map 2022-03-10T01:56:27.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:27.463783 W | op-mon: monitor c is not in quorum list 2022-03-10T01:56:28.662 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:56:29.469 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:56:29.470 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:56:29.470 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:56:29.470 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:56:29.470 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:56:29.471 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:56:29.471 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:56:29.471 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:56:29.471 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:56:29.472 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:56:29.472 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:56:29.472 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:56:29.472 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:56:29.473 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:56:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:56:32.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:32.483855 I | op-mon: mon c is not yet running 2022-03-10T01:56:32.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:32.483888 I | op-mon: mons running: [a b] 2022-03-10T01:56:37.495 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:37.509737 I | op-mon: mons running: [a b c] 2022-03-10T01:56:39.477 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:56:40.642 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:56:40.643 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:56:40.643 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:56:40.643 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:56:40.644 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:56:40.644 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:56:40.644 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:56:40.644 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:56:40.644 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:56:40.645 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:56:40.645 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:56:40.645 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:56:40.645 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:56:40.646 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:56:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:56:40.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:40.745176 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T01:56:40.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:40.745215 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:56:40.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:40.955283 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:56:40.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:40.955317 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:56:41.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:41.161761 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:56:41.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:41.161797 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:56:41.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:41.368527 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:56:41.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:41.368572 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:56:41.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:41.581362 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:56:41.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:41.581391 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:56:41.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:41.801824 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:56:41.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:41.801865 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T01:56:42.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:42.131625 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T01:56:42.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:42.463063 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T01:56:42.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:42.799959 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T01:56:43.230 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:43.244481 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T01:56:43.230 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:43.244507 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T01:56:43.588 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:43.603068 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T01:56:43.816 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:43.830735 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T01:56:43.816 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:43.830771 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T01:56:43.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:43.830780 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T01:56:44.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:44.376719 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T01:56:44.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:44.393538 I | op-mgr: start running mgr 2022-03-10T01:56:44.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:44.393569 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T01:56:45.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:45.152931 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi184": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:56:45.235 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:45.249195 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi184": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:56:50.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:50.562992 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi071": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:56:50.650 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:56:51.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:56:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-10T01:56:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-10T01:56:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-10T01:56:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-10T01:56:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-10T01:56:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-10T01:56:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-10T01:56:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-10T01:56:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-10T01:56:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-10T01:56:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-10T01:56:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-10T01:56:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-10T01:56:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-10T01:56:58.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:56:58.591072 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi007": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:57:01.225 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:57:01.978 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:57:01.981 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-10T01:57:01.986 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 2 2022-03-10T01:57:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-10T01:57:05.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.008075 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-10T01:57:05.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.011820 I | op-mgr: setting services to point to mgr "a" 2022-03-10T01:57:06.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.300154 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T01:57:06.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.300239 I | op-mgr: successful modules: balancer 2022-03-10T01:57:06.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.319329 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T01:57:06.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.319374 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-10T01:57:06.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.319388 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T01:57:06.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.553451 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T01:57:06.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.719096 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-10T01:57:06.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.719128 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T01:57:06.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.719136 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T01:57:06.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:06.893783 I | op-mgr: successful modules: dashboard 2022-03-10T01:57:07.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:07.189084 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T01:57:07.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:07.189118 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:57:07.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:07.200662 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-10T01:57:07.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:07.200695 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-10T01:57:07.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:07.200706 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-10T01:57:08.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:08.290263 I | op-mgr: successful modules: prometheus 2022-03-10T01:57:08.278 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:08.292463 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T01:57:11.990 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:57:16.773 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:57:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-10T01:57:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 2 2022-03-10T01:57:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-10T01:57:16.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:16.937937 I | op-mgr: successful modules: orchestrator modules 2022-03-10T01:57:26.784 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status -f json 2022-03-10T01:57:27.451 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:57:27.452 INFO:teuthology.orchestra.run.smithi007.stdout:{"available": true, "backend": "rook"} 2022-03-10T01:57:27.471 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-10T01:57:27.472 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-10T01:57:28.137 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch apply osd --all-available-devices 2022-03-10T01:57:28.971 INFO:teuthology.orchestra.run.smithi007.stdout:['Success'] 2022-03-10T01:57:28.972 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:28.985852 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T01:57:28.973 INFO:tasks.rook.operator.smithi007.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T01:57:28.973 INFO:tasks.rook.operator.smithi007.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T01:57:28.973 INFO:tasks.rook.operator.smithi007.stdout: ... // 3 identical fields 2022-03-10T01:57:28.973 INFO:tasks.rook.operator.smithi007.stdout: Config: nil, 2022-03-10T01:57:28.973 INFO:tasks.rook.operator.smithi007.stdout: Selection: {}, 2022-03-10T01:57:28.974 INFO:tasks.rook.operator.smithi007.stdout:- StorageClassDeviceSets: nil, 2022-03-10T01:57:28.974 INFO:tasks.rook.operator.smithi007.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T01:57:28.974 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.974 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi007-lv-1", 2022-03-10T01:57:28.980 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.981 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.981 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.982 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.982 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi007-lv-2", 2022-03-10T01:57:28.983 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.983 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.983 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.983 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.984 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi007-lv-3", 2022-03-10T01:57:28.984 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.984 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.985 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.985 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.986 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi007-lv-4", 2022-03-10T01:57:28.986 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.986 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.987 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.987 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.987 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi071-lv-1", 2022-03-10T01:57:28.988 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.988 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.989 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.989 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.990 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi071-lv-2", 2022-03-10T01:57:28.990 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.990 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.991 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.991 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.992 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi071-lv-3", 2022-03-10T01:57:28.992 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.993 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.993 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.993 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.994 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi071-lv-4", 2022-03-10T01:57:28.994 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.995 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.995 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.996 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.996 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi184-lv-1", 2022-03-10T01:57:28.996 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.997 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.997 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.997 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.998 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi184-lv-2", 2022-03-10T01:57:28.998 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.998 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:28.998 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:28.999 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:28.999 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi184-lv-3", 2022-03-10T01:57:28.999 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:28.999 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:29.000 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:29.000 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:57:29.000 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi184-lv-4", 2022-03-10T01:57:29.000 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:57:29.000 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-10T01:57:29.001 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:29.001 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:57:29.001 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-10T01:57:29.001 INFO:tasks.rook.operator.smithi007.stdout: Annotations: nil, 2022-03-10T01:57:29.002 INFO:tasks.rook.operator.smithi007.stdout: Labels: nil, 2022-03-10T01:57:29.002 INFO:tasks.rook.operator.smithi007.stdout: ... // 20 identical fields 2022-03-10T01:57:29.002 INFO:tasks.rook.operator.smithi007.stdout: } 2022-03-10T01:57:29.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:28.985969 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T01:57:29.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:28.986536 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-10T01:57:29.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:28.986575 I | operator: successfully started the controller-runtime manager 2022-03-10T01:57:29.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:28.986606 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-10T01:57:29.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:28.986620 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-10T01:57:29.003 INFO:tasks.rook.operator.smithi007.stdout:I0310 01:57:28.986664 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T01:57:29.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.008429 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T01:57:29.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.008455 I | operator: watching all namespaces for Ceph CRs 2022-03-10T01:57:29.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.008496 I | operator: setting up schemes 2022-03-10T01:57:29.006 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-10T01:57:29.006 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:57:29.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.012410 I | operator: setting up the controller-runtime manager 2022-03-10T01:57:29.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.215466 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T01:57:29.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218132 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T01:57:29.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218173 I | ceph-cluster-controller: successfully started 2022-03-10T01:57:29.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218230 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T01:57:29.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218249 I | ceph-crashcollector-controller: successfully started 2022-03-10T01:57:29.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218275 I | ceph-block-pool-controller: successfully started 2022-03-10T01:57:29.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218309 I | ceph-object-store-user-controller: successfully started 2022-03-10T01:57:29.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218340 I | ceph-object-realm-controller: successfully started 2022-03-10T01:57:29.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218363 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T01:57:29.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218374 I | ceph-object-zone-controller: successfully started 2022-03-10T01:57:29.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218516 I | ceph-object-controller: successfully started 2022-03-10T01:57:29.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218550 I | ceph-file-controller: successfully started 2022-03-10T01:57:29.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218574 I | ceph-nfs-controller: successfully started 2022-03-10T01:57:29.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218601 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T01:57:29.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218647 I | ceph-client-controller: successfully started 2022-03-10T01:57:29.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218665 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T01:57:29.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218693 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T01:57:29.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218707 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T01:57:29.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218720 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T01:57:29.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218735 I | ceph-bucket-topic: successfully started 2022-03-10T01:57:29.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218749 I | ceph-bucket-notification: successfully started 2022-03-10T01:57:29.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218763 I | ceph-bucket-notification: successfully started 2022-03-10T01:57:29.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.218774 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T01:57:29.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.219829 I | operator: starting the controller-runtime manager 2022-03-10T01:57:29.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.323344 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T01:57:29.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.323415 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T01:57:29.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.323434 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T01:57:29.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.325167 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:57:29.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.326645 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T01:57:29.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.327159 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:57:29.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.327213 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T01:57:29.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.327226 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:57:29.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.328851 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T01:57:29.316 INFO:tasks.rook.operator.smithi007.stdout:I0310 01:57:29.328919 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:57:29.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.329602 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:57:29.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.443974 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T01:57:29.434 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.446719 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:57:29.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.449953 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T01:57:29.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.449969 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T01:57:29.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.449974 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T01:57:29.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.449979 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T01:57:29.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.449984 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T01:57:29.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.449989 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T01:57:29.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.449995 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T01:57:29.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450001 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T01:57:29.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450006 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T01:57:29.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450012 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T01:57:29.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450019 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T01:57:29.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450025 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T01:57:29.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450031 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T01:57:29.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450035 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T01:57:29.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450040 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T01:57:29.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450045 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T01:57:29.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450084 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:57:29.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:29.450097 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:57:29.546 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:57:29.564 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:57:29.564 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T01:57:34.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986597 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T01:57:34.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986638 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T01:57:34.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986653 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:57:34.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986664 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:57:34.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986675 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:57:34.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986686 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:57:34.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986698 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:57:34.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986710 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:57:34.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986728 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T01:57:34.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986741 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T01:57:34.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986754 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:57:34.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986768 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:57:34.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986789 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:57:34.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986801 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:57:34.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986814 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T01:57:34.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986826 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T01:57:34.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986838 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T01:57:34.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986850 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T01:57:34.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986862 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T01:57:34.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986876 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:57:34.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986890 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:57:34.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986903 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T01:57:34.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986914 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T01:57:34.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986930 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T01:57:34.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:34.986942 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T01:57:35.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.025182 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T01:57:35.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.035242 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:57:35.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.035271 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:57:35.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.035277 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:57:35.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.035284 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:57:35.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.035292 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:57:35.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.035299 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:57:35.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.035306 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T01:57:35.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.224131 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:57:35.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.224167 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:57:35.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.224180 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:57:35.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.262183 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T01:57:35.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.457770 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:57:35.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.457799 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:57:35.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.457808 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T01:57:35.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.522028 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:57:35.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.522063 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:57:35.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.522071 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:57:35.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.563462 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:57:35.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.563489 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:57:35.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.563506 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:57:35.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:35.576955 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:57:36.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.026725 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:57:36.014 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.027029 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:57:36.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.027102 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T01:57:36.364 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.379628 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:57:36.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.384973 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T01:57:36.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.384998 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:57:36.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.389513 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T01:57:36.419 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.434436 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:57:36.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.711529 I | op-mon: start running mons 2022-03-10T01:57:36.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:36.737425 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:57:37.054 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:37.069261 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789","10.101.240.174:6789","10.111.0.148:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-10T01:57:37.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:37.667678 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:57:37.657 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:37.667976 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:57:39.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:39.268517 I | op-mon: targeting the mon count 3 2022-03-10T01:57:39.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:39.274431 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:57:39.566 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:57:39.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:39.614570 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:57:39.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:39.614601 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:57:39.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:40.009954 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:57:39.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:40.009978 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:57:40.078 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:57:40.098 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:57:40.099 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T01:57:40.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:40.343117 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:57:40.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:40.343151 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:57:40.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:40.781451 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T01:57:40.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:40.781487 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T01:57:41.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:41.125476 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T01:57:41.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:41.125507 I | op-mon: checking for basic quorum with existing mons 2022-03-10T01:57:41.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:41.304863 I | op-mon: mon "a" endpoint is [v2:10.99.228.129:3300,v1:10.99.228.129:6789] 2022-03-10T01:57:41.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:41.438789 I | op-mon: mon "b" endpoint is [v2:10.101.240.174:3300,v1:10.101.240.174:6789] 2022-03-10T01:57:41.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:41.669987 I | op-mon: mon "c" endpoint is [v2:10.111.0.148:3300,v1:10.111.0.148:6789] 2022-03-10T01:57:42.252 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:42.267577 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789","10.101.240.174:6789","10.111.0.148:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-10T01:57:42.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:42.867885 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:57:42.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:42.868161 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:57:43.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:43.309827 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T01:57:43.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:43.320142 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T01:57:43.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:43.320170 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T01:57:43.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:43.872633 I | op-mon: mons running: [a b c] 2022-03-10T01:57:44.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.320388 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T01:57:44.314 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.329171 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T01:57:44.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.339547 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T01:57:44.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.339570 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T01:57:44.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.471914 I | op-mon: mons running: [a b c] 2022-03-10T01:57:44.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.890078 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T01:57:44.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.902108 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T01:57:44.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.912156 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T01:57:44.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:44.912180 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T01:57:45.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:45.070764 I | op-mon: mons running: [a b c] 2022-03-10T01:57:45.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:45.476009 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T01:57:45.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:45.476037 I | op-mon: mons created: 3 2022-03-10T01:57:45.862 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:45.876912 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T01:57:45.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:45.901406 I | op-mon: mons running: [a b c] 2022-03-10T01:57:46.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:46.360276 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T01:57:46.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:46.360328 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T01:57:46.761 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:46.776239 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T01:57:47.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:47.206721 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T01:57:47.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:47.702544 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T01:57:48.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:48.166068 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T01:57:48.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:48.166096 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T01:57:48.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:48.651286 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T01:57:49.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:49.046691 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T01:57:49.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:49.046731 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T01:57:49.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:49.046742 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T01:57:49.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:49.470687 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T01:57:49.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:49.503036 I | op-mgr: start running mgr 2022-03-10T01:57:49.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:49.503072 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T01:57:50.082 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.097259 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T01:57:50.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.106738 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T01:57:50.094 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.109929 I | op-mgr: setting services to point to mgr "a" 2022-03-10T01:57:50.100 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:57:50.181 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.196028 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T01:57:50.181 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.196145 I | op-mgr: successful modules: balancer 2022-03-10T01:57:50.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.243192 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T01:57:50.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.243224 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T01:57:50.244 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.259247 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T01:57:50.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.340278 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-1" 2022-03-10T01:57:50.527 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.542732 I | op-osd: successfully provisioned PVC "smithi007-lv-1-data-0jzl5l" 2022-03-10T01:57:50.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.542761 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-2" 2022-03-10T01:57:50.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:57:50.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.742838 I | op-osd: successfully provisioned PVC "smithi007-lv-2-data-08blxh" 2022-03-10T01:57:50.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.742868 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-3" 2022-03-10T01:57:50.740 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:57:50.741 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T01:57:50.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.895565 I | op-mgr: successful modules: prometheus 2022-03-10T01:57:50.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.941971 I | op-osd: successfully provisioned PVC "smithi007-lv-3-data-0xslw4" 2022-03-10T01:57:50.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:50.942013 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-4" 2022-03-10T01:57:51.127 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.142105 I | op-osd: successfully provisioned PVC "smithi007-lv-4-data-0cfg7m" 2022-03-10T01:57:51.127 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.142142 I | op-osd: creating 1 new PVCs for device set "smithi071-lv-1" 2022-03-10T01:57:51.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.380392 I | op-osd: successfully provisioned PVC "smithi071-lv-1-data-0nctwr" 2022-03-10T01:57:51.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.380439 I | op-osd: creating 1 new PVCs for device set "smithi071-lv-2" 2022-03-10T01:57:51.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.541769 I | op-osd: successfully provisioned PVC "smithi071-lv-2-data-0lmhm6" 2022-03-10T01:57:51.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.541804 I | op-osd: creating 1 new PVCs for device set "smithi071-lv-3" 2022-03-10T01:57:51.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.741872 I | op-osd: successfully provisioned PVC "smithi071-lv-3-data-0xjgfz" 2022-03-10T01:57:51.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.741910 I | op-osd: creating 1 new PVCs for device set "smithi071-lv-4" 2022-03-10T01:57:51.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.947164 I | op-osd: successfully provisioned PVC "smithi071-lv-4-data-0q5d9z" 2022-03-10T01:57:51.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:51.947201 I | op-osd: creating 1 new PVCs for device set "smithi184-lv-1" 2022-03-10T01:57:52.082 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.097070 I | op-mgr: successful modules: dashboard 2022-03-10T01:57:52.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.138178 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T01:57:52.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.141788 I | op-osd: successfully provisioned PVC "smithi184-lv-1-data-096cgx" 2022-03-10T01:57:52.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.141833 I | op-osd: creating 1 new PVCs for device set "smithi184-lv-2" 2022-03-10T01:57:52.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.342385 I | op-osd: successfully provisioned PVC "smithi184-lv-2-data-0l5x8k" 2022-03-10T01:57:52.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.342418 I | op-osd: creating 1 new PVCs for device set "smithi184-lv-3" 2022-03-10T01:57:52.527 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.542664 I | op-osd: successfully provisioned PVC "smithi184-lv-3-data-09sg48" 2022-03-10T01:57:52.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.542696 I | op-osd: creating 1 new PVCs for device set "smithi184-lv-4" 2022-03-10T01:57:52.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.747139 I | op-osd: successfully provisioned PVC "smithi184-lv-4-data-0n7czp" 2022-03-10T01:57:52.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:52.758032 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0jzl5l" 2022-03-10T01:57:53.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:53.169674 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-1-data-0jzl5l" 2022-03-10T01:57:53.155 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:53.169707 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-08blxh" 2022-03-10T01:57:53.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:53.570633 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-2-data-08blxh" 2022-03-10T01:57:53.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:53.570669 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0xslw4" 2022-03-10T01:57:53.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:53.677482 I | op-mgr: successful modules: orchestrator modules 2022-03-10T01:57:53.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:53.963245 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-3-data-0xslw4" 2022-03-10T01:57:53.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:53.963283 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0cfg7m" 2022-03-10T01:57:54.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:54.853878 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-4-data-0cfg7m" 2022-03-10T01:57:54.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:54.853915 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-1-data-0nctwr" 2022-03-10T01:57:55.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:55.736349 I | op-osd: started OSD provisioning job for PVC "smithi071-lv-1-data-0nctwr" 2022-03-10T01:57:55.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:55.736384 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-2-data-0lmhm6" 2022-03-10T01:57:56.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:56.758985 I | op-osd: started OSD provisioning job for PVC "smithi071-lv-2-data-0lmhm6" 2022-03-10T01:57:56.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:56.759026 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-3-data-0xjgfz" 2022-03-10T01:57:57.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:57.567622 I | op-osd: started OSD provisioning job for PVC "smithi071-lv-3-data-0xjgfz" 2022-03-10T01:57:57.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:57.567655 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-4-data-0q5d9z" 2022-03-10T01:57:58.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:58.337676 I | op-osd: started OSD provisioning job for PVC "smithi071-lv-4-data-0q5d9z" 2022-03-10T01:57:58.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:58.337703 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-1-data-096cgx" 2022-03-10T01:57:59.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:59.624361 I | op-osd: started OSD provisioning job for PVC "smithi184-lv-1-data-096cgx" 2022-03-10T01:57:59.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:57:59.624402 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-2-data-0l5x8k" 2022-03-10T01:58:00.742 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:58:00.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:00.787767 I | op-osd: started OSD provisioning job for PVC "smithi184-lv-2-data-0l5x8k" 2022-03-10T01:58:00.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:00.787816 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-3-data-09sg48" 2022-03-10T01:58:02.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:02.374519 I | op-osd: started OSD provisioning job for PVC "smithi184-lv-3-data-09sg48" 2022-03-10T01:58:02.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:02.374552 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-4-data-0n7czp" 2022-03-10T01:58:02.876 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:58:02.897 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:58:02.897 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T01:58:03.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.273355 I | op-osd: started OSD provisioning job for PVC "smithi184-lv-4-data-0n7czp" 2022-03-10T01:58:03.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.273380 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T01:58:03.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.273385 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T01:58:03.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.798986 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-0jzl5l is "starting" 2022-03-10T01:58:03.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799010 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-08blxh is "starting" 2022-03-10T01:58:03.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799016 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-0xslw4 is "starting" 2022-03-10T01:58:03.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799021 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0cfg7m is "starting" 2022-03-10T01:58:03.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799025 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0nctwr is "starting" 2022-03-10T01:58:03.786 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799029 I | op-osd: OSD orchestration status for PVC smithi071-lv-2-data-0lmhm6 is "starting" 2022-03-10T01:58:03.786 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799033 I | op-osd: OSD orchestration status for PVC smithi071-lv-3-data-0xjgfz is "starting" 2022-03-10T01:58:03.786 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799043 I | op-osd: OSD orchestration status for PVC smithi071-lv-4-data-0q5d9z is "starting" 2022-03-10T01:58:03.786 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799047 I | op-osd: OSD orchestration status for PVC smithi184-lv-1-data-096cgx is "starting" 2022-03-10T01:58:03.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799051 I | op-osd: OSD orchestration status for PVC smithi184-lv-2-data-0l5x8k is "starting" 2022-03-10T01:58:03.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799056 I | op-osd: OSD orchestration status for PVC smithi184-lv-3-data-09sg48 is "starting" 2022-03-10T01:58:03.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:03.799063 I | op-osd: OSD orchestration status for PVC smithi184-lv-4-data-0n7czp is "starting" 2022-03-10T01:58:12.899 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:58:13.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:13.485116 I | op-osd: OSD orchestration status for node smithi071-lv-2-data-0lmhm6 is "orchestrating" 2022-03-10T01:58:13.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:13.485146 I | op-osd: OSD orchestration status for node smithi071-lv-1-data-0nctwr is "orchestrating" 2022-03-10T01:58:13.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:13.995361 I | op-osd: OSD orchestration status for PVC smithi071-lv-2-data-0lmhm6 is "orchestrating" 2022-03-10T01:58:14.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:14.206272 I | op-osd: OSD orchestration status for node smithi071-lv-3-data-0xjgfz is "orchestrating" 2022-03-10T01:58:14.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:14.206295 I | op-osd: OSD orchestration status for node smithi071-lv-4-data-0q5d9z is "orchestrating" 2022-03-10T01:58:14.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:14.355641 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0nctwr is "orchestrating" 2022-03-10T01:58:14.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:14.622906 I | op-osd: OSD orchestration status for node smithi007-lv-2-data-08blxh is "orchestrating" 2022-03-10T01:58:14.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:14.781443 I | op-osd: OSD orchestration status for PVC smithi071-lv-4-data-0q5d9z is "orchestrating" 2022-03-10T01:58:15.069 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:58:15.093 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-10T01:58:15.093 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T01:58:15.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:15.171718 I | op-osd: OSD orchestration status for PVC smithi071-lv-3-data-0xjgfz is "orchestrating" 2022-03-10T01:58:15.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:15.270600 I | op-osd: OSD orchestration status for node smithi007-lv-1-data-0jzl5l is "orchestrating" 2022-03-10T01:58:15.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:15.509507 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-08blxh is "orchestrating" 2022-03-10T01:58:16.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:16.411402 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-0jzl5l is "orchestrating" 2022-03-10T01:58:18.135 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:18.150886 I | op-osd: OSD orchestration status for node smithi007-lv-3-data-0xslw4 is "orchestrating" 2022-03-10T01:58:18.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:18.509162 I | op-osd: OSD orchestration status for node smithi007-lv-4-data-0cfg7m is "orchestrating" 2022-03-10T01:58:18.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:18.654174 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-0xslw4 is "orchestrating" 2022-03-10T01:58:18.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:18.878615 I | op-osd: OSD orchestration status for node smithi184-lv-2-data-0l5x8k is "orchestrating" 2022-03-10T01:58:18.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:18.905209 I | op-osd: OSD orchestration status for node smithi184-lv-3-data-09sg48 is "orchestrating" 2022-03-10T01:58:18.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:18.920629 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0cfg7m is "orchestrating" 2022-03-10T01:58:18.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:18.944150 I | op-osd: OSD orchestration status for node smithi184-lv-1-data-096cgx is "orchestrating" 2022-03-10T01:58:19.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:19.286893 I | op-osd: OSD orchestration status for node smithi184-lv-4-data-0n7czp is "orchestrating" 2022-03-10T01:58:19.314 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:19.328870 I | op-osd: OSD orchestration status for PVC smithi184-lv-2-data-0l5x8k is "orchestrating" 2022-03-10T01:58:19.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:19.384036 I | op-osd: OSD orchestration status for PVC smithi184-lv-3-data-09sg48 is "orchestrating" 2022-03-10T01:58:19.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:19.389531 I | op-osd: OSD orchestration status for PVC smithi184-lv-1-data-096cgx is "orchestrating" 2022-03-10T01:58:19.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:19.733373 I | op-osd: OSD orchestration status for PVC smithi184-lv-4-data-0n7czp is "orchestrating" 2022-03-10T01:58:22.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.275617 I | op-osd: OSD orchestration status for PVC smithi071-lv-2-data-0lmhm6 is "completed" 2022-03-10T01:58:22.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.275637 I | op-osd: creating OSD 1 on PVC "smithi071-lv-2-data-0lmhm6" 2022-03-10T01:58:22.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.275644 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-2-data-0lmhm6" 2022-03-10T01:58:22.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.456016 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0nctwr is "completed" 2022-03-10T01:58:22.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.456058 I | op-osd: creating OSD 0 on PVC "smithi071-lv-1-data-0nctwr" 2022-03-10T01:58:22.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.456073 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-1-data-0nctwr" 2022-03-10T01:58:22.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.758474 I | op-osd: OSD orchestration status for PVC smithi071-lv-3-data-0xjgfz is "completed" 2022-03-10T01:58:22.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.758512 I | op-osd: creating OSD 2 on PVC "smithi071-lv-3-data-0xjgfz" 2022-03-10T01:58:22.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:22.758529 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-3-data-0xjgfz" 2022-03-10T01:58:23.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:23.660882 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-08blxh is "completed" 2022-03-10T01:58:23.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:23.660926 I | op-osd: creating OSD 4 on PVC "smithi007-lv-2-data-08blxh" 2022-03-10T01:58:23.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:23.660945 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-08blxh" 2022-03-10T01:58:24.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:24.768615 I | op-osd: OSD orchestration status for PVC smithi071-lv-4-data-0q5d9z is "completed" 2022-03-10T01:58:24.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:24.768664 I | op-osd: creating OSD 3 on PVC "smithi071-lv-4-data-0q5d9z" 2022-03-10T01:58:24.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:24.768688 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-4-data-0q5d9z" 2022-03-10T01:58:25.095 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:58:26.306 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:58:26.331 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646877501,"num_remapped_pgs":0} 2022-03-10T01:58:26.331 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T01:58:26.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:26.454619 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-0jzl5l is "completed" 2022-03-10T01:58:26.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:26.454679 I | op-osd: creating OSD 5 on PVC "smithi007-lv-1-data-0jzl5l" 2022-03-10T01:58:26.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:26.454691 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0jzl5l" 2022-03-10T01:58:28.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:28.763788 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-0xslw4 is "completed" 2022-03-10T01:58:28.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:28.763813 I | op-osd: creating OSD 6 on PVC "smithi007-lv-3-data-0xslw4" 2022-03-10T01:58:28.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:28.763820 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0xslw4" 2022-03-10T01:58:29.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:29.961229 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0cfg7m is "completed" 2022-03-10T01:58:29.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:29.961267 I | op-osd: creating OSD 7 on PVC "smithi007-lv-4-data-0cfg7m" 2022-03-10T01:58:29.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:29.961280 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0cfg7m" 2022-03-10T01:58:32.453 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:32.463736 I | op-osd: OSD orchestration status for PVC smithi184-lv-1-data-096cgx is "completed" 2022-03-10T01:58:32.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:32.463779 I | op-osd: creating OSD 9 on PVC "smithi184-lv-1-data-096cgx" 2022-03-10T01:58:32.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:32.463796 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-1-data-096cgx" 2022-03-10T01:58:34.475 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:34.484060 I | op-osd: OSD orchestration status for PVC smithi184-lv-2-data-0l5x8k is "completed" 2022-03-10T01:58:34.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:34.484123 I | op-osd: creating OSD 8 on PVC "smithi184-lv-2-data-0l5x8k" 2022-03-10T01:58:34.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:34.484150 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-2-data-0l5x8k" 2022-03-10T01:58:35.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:35.824309 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi184": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:58:36.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:36.178207 I | op-osd: OSD orchestration status for PVC smithi184-lv-3-data-09sg48 is "completed" 2022-03-10T01:58:36.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:36.178235 I | op-osd: creating OSD 10 on PVC "smithi184-lv-3-data-09sg48" 2022-03-10T01:58:36.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:36.178251 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-3-data-09sg48" 2022-03-10T01:58:36.332 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:58:37.866 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:58:37.889 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646877501,"num_remapped_pgs":0} 2022-03-10T01:58:37.890 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T01:58:38.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:38.200336 I | op-osd: OSD orchestration status for PVC smithi184-lv-4-data-0n7czp is "completed" 2022-03-10T01:58:38.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:38.200373 I | op-osd: creating OSD 11 on PVC "smithi184-lv-4-data-0n7czp" 2022-03-10T01:58:38.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:38.200396 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-4-data-0n7czp" 2022-03-10T01:58:41.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:41.964988 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T01:58:41.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:58:41.965024 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:58:47.892 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:58:51.010 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:58:51.033 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646877501,"num_remapped_pgs":0} 2022-03-10T01:58:51.033 INFO:tasks.rook: have 0/12 OSDs 2022-03-10T01:59:01.034 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:59:02.147 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:02.171 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"num_osds":12,"num_up_osds":4,"osd_up_since":1646877538,"num_in_osds":12,"osd_in_since":1646877501,"num_remapped_pgs":0} 2022-03-10T01:59:02.171 INFO:tasks.rook: have 4/12 OSDs 2022-03-10T01:59:12.173 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd stat -f json 2022-03-10T01:59:12.644 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:12.664 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":31,"num_osds":12,"num_up_osds":12,"osd_up_since":1646877550,"num_in_osds":12,"osd_in_since":1646877501,"num_remapped_pgs":0} 2022-03-10T01:59:12.665 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-10T01:59:12.665 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- cat /etc/ceph/ceph.conf 2022-03-10T01:59:12.889 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2022-03-10T01:59:12.890 INFO:teuthology.orchestra.run.smithi007.stdout:mon_host = 10.99.228.129:6789,10.101.240.174:6789,10.111.0.148:6789 2022-03-10T01:59:12.890 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:12.890 INFO:teuthology.orchestra.run.smithi007.stdout:[client.admin] 2022-03-10T01:59:12.890 INFO:teuthology.orchestra.run.smithi007.stdout:keyring = /etc/ceph/keyring 2022-03-10T01:59:12.894 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- cat /etc/ceph/keyring 2022-03-10T01:59:13.083 INFO:teuthology.orchestra.run.smithi007.stdout:[client.admin] 2022-03-10T01:59:13.084 INFO:teuthology.orchestra.run.smithi007.stdout:key = AQBGWiliKk45CxAAS+7iDXYgr0Jz2CEGjU3ByQ== 2022-03-10T01:59:13.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2022-03-10T01:59:13.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph 2022-03-10T01:59:13.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2022-03-10T01:59:13.135 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:59:13.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T01:59:13.157 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:59:13.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/keyring 2022-03-10T01:59:13.171 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:59:13.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T01:59:13.187 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:59:13.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/keyring 2022-03-10T01:59:13.215 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:59:13.215 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.conf 2022-03-10T01:59:13.229 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:59:13.229 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/keyring 2022-03-10T01:59:13.286 INFO:tasks.rook:Setting up client nodes... 2022-03-10T01:59:13.287 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T01:59:14.008 INFO:teuthology.orchestra.run.smithi007.stdout:[client.a] 2022-03-10T01:59:14.008 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBxWyliI1STNhAA8yOeaVTRO7fTDXmRl5Hl5w== 2022-03-10T01:59:14.031 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-10T01:59:14.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-10T01:59:14.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-10T01:59:14.056 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T01:59:14.767 INFO:teuthology.orchestra.run.smithi007.stdout:[client.b] 2022-03-10T01:59:14.768 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQByWylibHp7JxAABjsWDy+h9fOdYe1QOC5ImA== 2022-03-10T01:59:14.790 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2022-03-10T01:59:14.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-10T01:59:14.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-10T01:59:14.817 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-10T01:59:15.551 INFO:teuthology.orchestra.run.smithi007.stdout:[client.c] 2022-03-10T01:59:15.551 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBzWylicdR7GRAAyNtW+1zMv3GZ6/Wh1c4d/g== 2022-03-10T01:59:15.579 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2022-03-10T01:59:15.580 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-10T01:59:15.580 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-10T01:59:15.601 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd dump --format=json 2022-03-10T01:59:16.124 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:16.125 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":32,"fsid":"699947c3-1b9d-45f5-8d1c-87f65be1d025","created":"2022-03-10T01:54:33.620449+0000","modified":"2022-03-10T01:59:13.034585+0000","last_up_change":"2022-03-10T01:59:10.740518+0000","last_in_change":"2022-03-10T01:58:21.464267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T01:59:00.875103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d5dba693-d9fe-4753-aa9c-612a19a49c3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6800","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6801","nonce":340002148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6802","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6803","nonce":340002148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6806","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6807","nonce":340002148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6804","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6805","nonce":340002148}]},"public_addr":"10.248.50.13:6801/340002148","cluster_addr":"10.248.50.13:6803/340002148","heartbeat_back_addr":"10.248.50.13:6807/340002148","heartbeat_front_addr":"10.248.50.13:6805/340002148","state":["exists","up"]},{"osd":1,"uuid":"199f7f80-0fa6-4e0d-b36f-b23c7304e02d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6800","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6801","nonce":1639952650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6802","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6803","nonce":1639952650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6806","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6807","nonce":1639952650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6804","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6805","nonce":1639952650}]},"public_addr":"10.248.50.14:6801/1639952650","cluster_addr":"10.248.50.14:6803/1639952650","heartbeat_back_addr":"10.248.50.14:6807/1639952650","heartbeat_front_addr":"10.248.50.14:6805/1639952650","state":["exists","up"]},{"osd":2,"uuid":"e105e659-aa75-4f36-bc44-090763339e96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6800","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6801","nonce":1150324230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6802","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6803","nonce":1150324230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6806","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6807","nonce":1150324230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6804","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6805","nonce":1150324230}]},"public_addr":"10.248.50.15:6801/1150324230","cluster_addr":"10.248.50.15:6803/1150324230","heartbeat_back_addr":"10.248.50.15:6807/1150324230","heartbeat_front_addr":"10.248.50.15:6805/1150324230","state":["exists","up"]},{"osd":3,"uuid":"69839e37-04c7-4980-8af5-f57e3207c9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6800","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6801","nonce":1915312444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6802","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6803","nonce":1915312444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6806","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6807","nonce":1915312444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6804","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6805","nonce":1915312444}]},"public_addr":"10.248.50.16:6801/1915312444","cluster_addr":"10.248.50.16:6803/1915312444","heartbeat_back_addr":"10.248.50.16:6807/1915312444","heartbeat_front_addr":"10.248.50.16:6805/1915312444","state":["exists","up"]},{"osd":4,"uuid":"0146e4ac-158a-4725-876d-c8a2dd5aeba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6800","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6801","nonce":87080672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6802","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6803","nonce":87080672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6806","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6807","nonce":87080672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6804","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6805","nonce":87080672}]},"public_addr":"10.248.48.11:6801/87080672","cluster_addr":"10.248.48.11:6803/87080672","heartbeat_back_addr":"10.248.48.11:6807/87080672","heartbeat_front_addr":"10.248.48.11:6805/87080672","state":["exists","up"]},{"osd":5,"uuid":"4fc4bb0e-ea71-4c04-8b1c-df934a080970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6800","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6801","nonce":978304682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6802","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6803","nonce":978304682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6806","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6807","nonce":978304682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6804","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6805","nonce":978304682}]},"public_addr":"10.248.48.12:6801/978304682","cluster_addr":"10.248.48.12:6803/978304682","heartbeat_back_addr":"10.248.48.12:6807/978304682","heartbeat_front_addr":"10.248.48.12:6805/978304682","state":["exists","up"]},{"osd":6,"uuid":"d3e721e3-f294-43c0-88c5-8f644b9de47a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6800","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6801","nonce":1433058284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6802","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6803","nonce":1433058284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6806","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6807","nonce":1433058284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6804","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6805","nonce":1433058284}]},"public_addr":"10.248.48.13:6801/1433058284","cluster_addr":"10.248.48.13:6803/1433058284","heartbeat_back_addr":"10.248.48.13:6807/1433058284","heartbeat_front_addr":"10.248.48.13:6805/1433058284","state":["exists","up"]},{"osd":7,"uuid":"e59705ae-5b02-4282-88a4-6e77e7a9c07d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6800","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6801","nonce":3327990237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6802","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6803","nonce":3327990237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6806","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6807","nonce":3327990237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6804","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6805","nonce":3327990237}]},"public_addr":"10.248.48.14:6801/3327990237","cluster_addr":"10.248.48.14:6803/3327990237","heartbeat_back_addr":"10.248.48.14:6807/3327990237","heartbeat_front_addr":"10.248.48.14:6805/3327990237","state":["exists","up"]},{"osd":8,"uuid":"af12001b-9379-4d82-98cf-d765dceb2f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6800","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6801","nonce":1008512955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6802","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6803","nonce":1008512955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6806","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6807","nonce":1008512955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6804","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6805","nonce":1008512955}]},"public_addr":"10.248.49.18:6801/1008512955","cluster_addr":"10.248.49.18:6803/1008512955","heartbeat_back_addr":"10.248.49.18:6807/1008512955","heartbeat_front_addr":"10.248.49.18:6805/1008512955","state":["exists","up"]},{"osd":9,"uuid":"3474ffc3-fc8b-4787-a59b-ad2b873bddcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6800","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6801","nonce":56556160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6802","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6803","nonce":56556160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6806","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6807","nonce":56556160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6804","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6805","nonce":56556160}]},"public_addr":"10.248.49.16:6801/56556160","cluster_addr":"10.248.49.16:6803/56556160","heartbeat_back_addr":"10.248.49.16:6807/56556160","heartbeat_front_addr":"10.248.49.16:6805/56556160","state":["exists","up"]},{"osd":10,"uuid":"9059c72a-1c4b-4d21-bb28-07afd05e7eba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6800","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6801","nonce":2345147718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6802","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6803","nonce":2345147718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6806","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6807","nonce":2345147718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6804","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6805","nonce":2345147718}]},"public_addr":"10.248.49.19:6801/2345147718","cluster_addr":"10.248.49.19:6803/2345147718","heartbeat_back_addr":"10.248.49.19:6807/2345147718","heartbeat_front_addr":"10.248.49.19:6805/2345147718","state":["exists","up"]},{"osd":11,"uuid":"a36a2483-4a26-41ea-a0e3-7d3c3fa42401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6800","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6801","nonce":1980570179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6802","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6803","nonce":1980570179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6806","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6807","nonce":1980570179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6804","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6805","nonce":1980570179}]},"public_addr":"10.248.49.20:6801/1980570179","cluster_addr":"10.248.49.20:6803/1980570179","heartbeat_back_addr":"10.248.49.20:6807/1980570179","heartbeat_front_addr":"10.248.49.20:6805/1980570179","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:50.283984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:53.303847+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:54.550453+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:53.175706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.364821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.106257+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.219694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.541358+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.554500+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:05.761155+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.576166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.570231+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.49.9:6801/1848651293":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/1567792517":"2022-03-11T01:57:14.999881+0000","10.248.49.9:6800/1848651293":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/3183879471":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/1792334852":"2022-03-11T01:57:14.999881+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T01:59:16.146 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-10T01:59:00.875103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-10T01:59:16.147 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd pool get .mgr pg_num 2022-03-10T01:59:16.736 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2022-03-10T01:59:16.756 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-10T01:59:16.757 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-10T01:59:16.757 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph mgr dump --format=json 2022-03-10T01:59:17.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:17.321 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"active_gid":14296,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.9:6800","nonce":968938479},{"type":"v1","addr":"10.248.49.9:6801","nonce":968938479}]},"active_addr":"10.248.49.9:6801/968938479","active_change":"2022-03-10T01:57:15.030944+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.248.49.9:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.248.49.9:0","nonce":2537012131}]},{"addrvec":[{"type":"v2","addr":"10.248.49.9:0","nonce":3073270621}]},{"addrvec":[{"type":"v2","addr":"10.248.49.9:0","nonce":4044371795}]},{"addrvec":[{"type":"v2","addr":"10.248.49.9:0","nonce":3209655157}]}]}} 2022-03-10T01:59:17.326 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-10T01:59:17.327 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-10T01:59:17.327 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd dump --format=json 2022-03-10T01:59:17.851 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:17.852 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":32,"fsid":"699947c3-1b9d-45f5-8d1c-87f65be1d025","created":"2022-03-10T01:54:33.620449+0000","modified":"2022-03-10T01:59:13.034585+0000","last_up_change":"2022-03-10T01:59:10.740518+0000","last_in_change":"2022-03-10T01:58:21.464267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T01:59:00.875103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d5dba693-d9fe-4753-aa9c-612a19a49c3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6800","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6801","nonce":340002148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6802","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6803","nonce":340002148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6806","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6807","nonce":340002148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6804","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6805","nonce":340002148}]},"public_addr":"10.248.50.13:6801/340002148","cluster_addr":"10.248.50.13:6803/340002148","heartbeat_back_addr":"10.248.50.13:6807/340002148","heartbeat_front_addr":"10.248.50.13:6805/340002148","state":["exists","up"]},{"osd":1,"uuid":"199f7f80-0fa6-4e0d-b36f-b23c7304e02d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6800","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6801","nonce":1639952650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6802","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6803","nonce":1639952650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6806","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6807","nonce":1639952650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6804","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6805","nonce":1639952650}]},"public_addr":"10.248.50.14:6801/1639952650","cluster_addr":"10.248.50.14:6803/1639952650","heartbeat_back_addr":"10.248.50.14:6807/1639952650","heartbeat_front_addr":"10.248.50.14:6805/1639952650","state":["exists","up"]},{"osd":2,"uuid":"e105e659-aa75-4f36-bc44-090763339e96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6800","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6801","nonce":1150324230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6802","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6803","nonce":1150324230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6806","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6807","nonce":1150324230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6804","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6805","nonce":1150324230}]},"public_addr":"10.248.50.15:6801/1150324230","cluster_addr":"10.248.50.15:6803/1150324230","heartbeat_back_addr":"10.248.50.15:6807/1150324230","heartbeat_front_addr":"10.248.50.15:6805/1150324230","state":["exists","up"]},{"osd":3,"uuid":"69839e37-04c7-4980-8af5-f57e3207c9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6800","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6801","nonce":1915312444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6802","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6803","nonce":1915312444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6806","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6807","nonce":1915312444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6804","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6805","nonce":1915312444}]},"public_addr":"10.248.50.16:6801/1915312444","cluster_addr":"10.248.50.16:6803/1915312444","heartbeat_back_addr":"10.248.50.16:6807/1915312444","heartbeat_front_addr":"10.248.50.16:6805/1915312444","state":["exists","up"]},{"osd":4,"uuid":"0146e4ac-158a-4725-876d-c8a2dd5aeba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6800","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6801","nonce":87080672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6802","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6803","nonce":87080672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6806","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6807","nonce":87080672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6804","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6805","nonce":87080672}]},"public_addr":"10.248.48.11:6801/87080672","cluster_addr":"10.248.48.11:6803/87080672","heartbeat_back_addr":"10.248.48.11:6807/87080672","heartbeat_front_addr":"10.248.48.11:6805/87080672","state":["exists","up"]},{"osd":5,"uuid":"4fc4bb0e-ea71-4c04-8b1c-df934a080970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6800","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6801","nonce":978304682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6802","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6803","nonce":978304682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6806","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6807","nonce":978304682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6804","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6805","nonce":978304682}]},"public_addr":"10.248.48.12:6801/978304682","cluster_addr":"10.248.48.12:6803/978304682","heartbeat_back_addr":"10.248.48.12:6807/978304682","heartbeat_front_addr":"10.248.48.12:6805/978304682","state":["exists","up"]},{"osd":6,"uuid":"d3e721e3-f294-43c0-88c5-8f644b9de47a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6800","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6801","nonce":1433058284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6802","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6803","nonce":1433058284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6806","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6807","nonce":1433058284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6804","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6805","nonce":1433058284}]},"public_addr":"10.248.48.13:6801/1433058284","cluster_addr":"10.248.48.13:6803/1433058284","heartbeat_back_addr":"10.248.48.13:6807/1433058284","heartbeat_front_addr":"10.248.48.13:6805/1433058284","state":["exists","up"]},{"osd":7,"uuid":"e59705ae-5b02-4282-88a4-6e77e7a9c07d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6800","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6801","nonce":3327990237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6802","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6803","nonce":3327990237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6806","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6807","nonce":3327990237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6804","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6805","nonce":3327990237}]},"public_addr":"10.248.48.14:6801/3327990237","cluster_addr":"10.248.48.14:6803/3327990237","heartbeat_back_addr":"10.248.48.14:6807/3327990237","heartbeat_front_addr":"10.248.48.14:6805/3327990237","state":["exists","up"]},{"osd":8,"uuid":"af12001b-9379-4d82-98cf-d765dceb2f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6800","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6801","nonce":1008512955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6802","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6803","nonce":1008512955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6806","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6807","nonce":1008512955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6804","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6805","nonce":1008512955}]},"public_addr":"10.248.49.18:6801/1008512955","cluster_addr":"10.248.49.18:6803/1008512955","heartbeat_back_addr":"10.248.49.18:6807/1008512955","heartbeat_front_addr":"10.248.49.18:6805/1008512955","state":["exists","up"]},{"osd":9,"uuid":"3474ffc3-fc8b-4787-a59b-ad2b873bddcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6800","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6801","nonce":56556160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6802","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6803","nonce":56556160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6806","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6807","nonce":56556160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6804","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6805","nonce":56556160}]},"public_addr":"10.248.49.16:6801/56556160","cluster_addr":"10.248.49.16:6803/56556160","heartbeat_back_addr":"10.248.49.16:6807/56556160","heartbeat_front_addr":"10.248.49.16:6805/56556160","state":["exists","up"]},{"osd":10,"uuid":"9059c72a-1c4b-4d21-bb28-07afd05e7eba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6800","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6801","nonce":2345147718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6802","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6803","nonce":2345147718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6806","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6807","nonce":2345147718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6804","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6805","nonce":2345147718}]},"public_addr":"10.248.49.19:6801/2345147718","cluster_addr":"10.248.49.19:6803/2345147718","heartbeat_back_addr":"10.248.49.19:6807/2345147718","heartbeat_front_addr":"10.248.49.19:6805/2345147718","state":["exists","up"]},{"osd":11,"uuid":"a36a2483-4a26-41ea-a0e3-7d3c3fa42401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6800","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6801","nonce":1980570179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6802","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6803","nonce":1980570179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6806","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6807","nonce":1980570179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6804","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6805","nonce":1980570179}]},"public_addr":"10.248.49.20:6801/1980570179","cluster_addr":"10.248.49.20:6803/1980570179","heartbeat_back_addr":"10.248.49.20:6807/1980570179","heartbeat_front_addr":"10.248.49.20:6805/1980570179","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:50.283984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:53.303847+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:54.550453+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:53.175706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.364821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.106257+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.219694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.541358+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.554500+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:05.761155+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.576166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.570231+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.49.9:6801/1848651293":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/1567792517":"2022-03-11T01:57:14.999881+0000","10.248.49.9:6800/1848651293":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/3183879471":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/1792334852":"2022-03-11T01:57:14.999881+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T01:59:17.871 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-10T01:59:17.872 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd dump --format=json 2022-03-10T01:59:18.457 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:18.458 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":32,"fsid":"699947c3-1b9d-45f5-8d1c-87f65be1d025","created":"2022-03-10T01:54:33.620449+0000","modified":"2022-03-10T01:59:13.034585+0000","last_up_change":"2022-03-10T01:59:10.740518+0000","last_in_change":"2022-03-10T01:58:21.464267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-10T01:59:00.875103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d5dba693-d9fe-4753-aa9c-612a19a49c3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6800","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6801","nonce":340002148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6802","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6803","nonce":340002148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6806","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6807","nonce":340002148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.13:6804","nonce":340002148},{"type":"v1","addr":"10.248.50.13:6805","nonce":340002148}]},"public_addr":"10.248.50.13:6801/340002148","cluster_addr":"10.248.50.13:6803/340002148","heartbeat_back_addr":"10.248.50.13:6807/340002148","heartbeat_front_addr":"10.248.50.13:6805/340002148","state":["exists","up"]},{"osd":1,"uuid":"199f7f80-0fa6-4e0d-b36f-b23c7304e02d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6800","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6801","nonce":1639952650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6802","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6803","nonce":1639952650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6806","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6807","nonce":1639952650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.14:6804","nonce":1639952650},{"type":"v1","addr":"10.248.50.14:6805","nonce":1639952650}]},"public_addr":"10.248.50.14:6801/1639952650","cluster_addr":"10.248.50.14:6803/1639952650","heartbeat_back_addr":"10.248.50.14:6807/1639952650","heartbeat_front_addr":"10.248.50.14:6805/1639952650","state":["exists","up"]},{"osd":2,"uuid":"e105e659-aa75-4f36-bc44-090763339e96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6800","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6801","nonce":1150324230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6802","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6803","nonce":1150324230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6806","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6807","nonce":1150324230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.15:6804","nonce":1150324230},{"type":"v1","addr":"10.248.50.15:6805","nonce":1150324230}]},"public_addr":"10.248.50.15:6801/1150324230","cluster_addr":"10.248.50.15:6803/1150324230","heartbeat_back_addr":"10.248.50.15:6807/1150324230","heartbeat_front_addr":"10.248.50.15:6805/1150324230","state":["exists","up"]},{"osd":3,"uuid":"69839e37-04c7-4980-8af5-f57e3207c9c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6800","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6801","nonce":1915312444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6802","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6803","nonce":1915312444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6806","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6807","nonce":1915312444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.16:6804","nonce":1915312444},{"type":"v1","addr":"10.248.50.16:6805","nonce":1915312444}]},"public_addr":"10.248.50.16:6801/1915312444","cluster_addr":"10.248.50.16:6803/1915312444","heartbeat_back_addr":"10.248.50.16:6807/1915312444","heartbeat_front_addr":"10.248.50.16:6805/1915312444","state":["exists","up"]},{"osd":4,"uuid":"0146e4ac-158a-4725-876d-c8a2dd5aeba3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6800","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6801","nonce":87080672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6802","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6803","nonce":87080672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6806","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6807","nonce":87080672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.11:6804","nonce":87080672},{"type":"v1","addr":"10.248.48.11:6805","nonce":87080672}]},"public_addr":"10.248.48.11:6801/87080672","cluster_addr":"10.248.48.11:6803/87080672","heartbeat_back_addr":"10.248.48.11:6807/87080672","heartbeat_front_addr":"10.248.48.11:6805/87080672","state":["exists","up"]},{"osd":5,"uuid":"4fc4bb0e-ea71-4c04-8b1c-df934a080970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6800","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6801","nonce":978304682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6802","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6803","nonce":978304682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6806","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6807","nonce":978304682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.12:6804","nonce":978304682},{"type":"v1","addr":"10.248.48.12:6805","nonce":978304682}]},"public_addr":"10.248.48.12:6801/978304682","cluster_addr":"10.248.48.12:6803/978304682","heartbeat_back_addr":"10.248.48.12:6807/978304682","heartbeat_front_addr":"10.248.48.12:6805/978304682","state":["exists","up"]},{"osd":6,"uuid":"d3e721e3-f294-43c0-88c5-8f644b9de47a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6800","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6801","nonce":1433058284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6802","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6803","nonce":1433058284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6806","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6807","nonce":1433058284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.13:6804","nonce":1433058284},{"type":"v1","addr":"10.248.48.13:6805","nonce":1433058284}]},"public_addr":"10.248.48.13:6801/1433058284","cluster_addr":"10.248.48.13:6803/1433058284","heartbeat_back_addr":"10.248.48.13:6807/1433058284","heartbeat_front_addr":"10.248.48.13:6805/1433058284","state":["exists","up"]},{"osd":7,"uuid":"e59705ae-5b02-4282-88a4-6e77e7a9c07d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6800","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6801","nonce":3327990237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6802","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6803","nonce":3327990237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6806","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6807","nonce":3327990237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.14:6804","nonce":3327990237},{"type":"v1","addr":"10.248.48.14:6805","nonce":3327990237}]},"public_addr":"10.248.48.14:6801/3327990237","cluster_addr":"10.248.48.14:6803/3327990237","heartbeat_back_addr":"10.248.48.14:6807/3327990237","heartbeat_front_addr":"10.248.48.14:6805/3327990237","state":["exists","up"]},{"osd":8,"uuid":"af12001b-9379-4d82-98cf-d765dceb2f01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6800","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6801","nonce":1008512955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6802","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6803","nonce":1008512955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6806","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6807","nonce":1008512955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.18:6804","nonce":1008512955},{"type":"v1","addr":"10.248.49.18:6805","nonce":1008512955}]},"public_addr":"10.248.49.18:6801/1008512955","cluster_addr":"10.248.49.18:6803/1008512955","heartbeat_back_addr":"10.248.49.18:6807/1008512955","heartbeat_front_addr":"10.248.49.18:6805/1008512955","state":["exists","up"]},{"osd":9,"uuid":"3474ffc3-fc8b-4787-a59b-ad2b873bddcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6800","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6801","nonce":56556160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6802","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6803","nonce":56556160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6806","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6807","nonce":56556160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.16:6804","nonce":56556160},{"type":"v1","addr":"10.248.49.16:6805","nonce":56556160}]},"public_addr":"10.248.49.16:6801/56556160","cluster_addr":"10.248.49.16:6803/56556160","heartbeat_back_addr":"10.248.49.16:6807/56556160","heartbeat_front_addr":"10.248.49.16:6805/56556160","state":["exists","up"]},{"osd":10,"uuid":"9059c72a-1c4b-4d21-bb28-07afd05e7eba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6800","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6801","nonce":2345147718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6802","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6803","nonce":2345147718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6806","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6807","nonce":2345147718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.19:6804","nonce":2345147718},{"type":"v1","addr":"10.248.49.19:6805","nonce":2345147718}]},"public_addr":"10.248.49.19:6801/2345147718","cluster_addr":"10.248.49.19:6803/2345147718","heartbeat_back_addr":"10.248.49.19:6807/2345147718","heartbeat_front_addr":"10.248.49.19:6805/2345147718","state":["exists","up"]},{"osd":11,"uuid":"a36a2483-4a26-41ea-a0e3-7d3c3fa42401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6800","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6801","nonce":1980570179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6802","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6803","nonce":1980570179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6806","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6807","nonce":1980570179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.49.20:6804","nonce":1980570179},{"type":"v1","addr":"10.248.49.20:6805","nonce":1980570179}]},"public_addr":"10.248.49.20:6801/1980570179","cluster_addr":"10.248.49.20:6803/1980570179","heartbeat_back_addr":"10.248.49.20:6807/1980570179","heartbeat_front_addr":"10.248.49.20:6805/1980570179","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:50.283984+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:53.303847+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:54.550453+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:58:53.175706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.364821+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.106257+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.219694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:04.541358+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.554500+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:05.761155+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.576166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-10T01:59:08.570231+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.49.9:6801/1848651293":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/1567792517":"2022-03-11T01:57:14.999881+0000","10.248.49.9:6800/1848651293":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/3183879471":"2022-03-11T01:57:14.999881+0000","10.248.49.9:0/1792334852":"2022-03-11T01:57:14.999881+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-10T01:59:18.477 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.0 flush_pg_stats 2022-03-10T01:59:18.479 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.1 flush_pg_stats 2022-03-10T01:59:18.480 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.2 flush_pg_stats 2022-03-10T01:59:18.481 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.3 flush_pg_stats 2022-03-10T01:59:18.482 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.4 flush_pg_stats 2022-03-10T01:59:18.486 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.5 flush_pg_stats 2022-03-10T01:59:18.487 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.6 flush_pg_stats 2022-03-10T01:59:18.488 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.7 flush_pg_stats 2022-03-10T01:59:18.488 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.8 flush_pg_stats 2022-03-10T01:59:18.489 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.9 flush_pg_stats 2022-03-10T01:59:18.490 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.10 flush_pg_stats 2022-03-10T01:59:18.491 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph tell osd.11 flush_pg_stats 2022-03-10T01:59:19.047 INFO:teuthology.orchestra.run.smithi007.stdout:115964116998 2022-03-10T01:59:19.048 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.9 2022-03-10T01:59:19.169 INFO:teuthology.orchestra.run.smithi007.stdout:115964116997 2022-03-10T01:59:19.169 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.7 2022-03-10T01:59:19.489 INFO:teuthology.orchestra.run.smithi007.stdout:73014444040 2022-03-10T01:59:19.489 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.0 2022-03-10T01:59:19.554 INFO:teuthology.orchestra.run.smithi007.stdout:115964116997 2022-03-10T01:59:19.555 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.6 2022-03-10T01:59:19.558 INFO:teuthology.orchestra.run.smithi007.stdout:128849018884 2022-03-10T01:59:19.559 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.8 2022-03-10T01:59:19.560 INFO:teuthology.orchestra.run.smithi007.stdout:85899345927 2022-03-10T01:59:19.561 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.2 2022-03-10T01:59:19.563 INFO:teuthology.orchestra.run.smithi007.stdout:81604378631 2022-03-10T01:59:19.563 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.1 2022-03-10T01:59:19.567 INFO:teuthology.orchestra.run.smithi007.stdout:90194313225 2022-03-10T01:59:19.568 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.3 2022-03-10T01:59:19.571 INFO:teuthology.orchestra.run.smithi007.stdout:115964116998 2022-03-10T01:59:19.572 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.4 2022-03-10T01:59:19.618 INFO:teuthology.orchestra.run.smithi007.stdout:128849018884 2022-03-10T01:59:19.618 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.11 2022-03-10T01:59:19.668 INFO:teuthology.orchestra.run.smithi007.stdout:115964116996 2022-03-10T01:59:19.668 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.5 2022-03-10T01:59:19.996 INFO:teuthology.orchestra.run.smithi007.stdout:115964116996 2022-03-10T01:59:20.041 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116996 for osd.9 2022-03-10T01:59:20.136 INFO:teuthology.orchestra.run.smithi007.stdout:115964116997 2022-03-10T01:59:20.138 INFO:teuthology.orchestra.run.smithi007.stdout:128849018884 2022-03-10T01:59:20.139 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.10 2022-03-10T01:59:20.179 INFO:teuthology.orchestra.run.smithi007.stdout:73014444040 2022-03-10T01:59:20.402 INFO:teuthology.orchestra.run.smithi007.stdout:128849018884 2022-03-10T01:59:20.505 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116997 for osd.7 2022-03-10T01:59:20.505 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.509 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444040 got 73014444040 for osd.0 2022-03-10T01:59:20.509 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.518 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018884 got 128849018884 for osd.8 2022-03-10T01:59:20.518 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.606 INFO:teuthology.orchestra.run.smithi007.stdout:81604378631 2022-03-10T01:59:20.659 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378631 got 81604378631 for osd.1 2022-03-10T01:59:20.659 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.777 INFO:teuthology.orchestra.run.smithi007.stdout:115964116997 2022-03-10T01:59:20.788 INFO:teuthology.orchestra.run.smithi007.stdout:85899345927 2022-03-10T01:59:20.807 INFO:teuthology.orchestra.run.smithi007.stdout:115964116995 2022-03-10T01:59:20.818 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116997 for osd.6 2022-03-10T01:59:20.819 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.820 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345927 got 85899345927 for osd.2 2022-03-10T01:59:20.820 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.831 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116996 got 115964116995 for osd.5 2022-03-10T01:59:20.844 INFO:teuthology.orchestra.run.smithi007.stdout:115964116998 2022-03-10T01:59:20.868 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116998 for osd.4 2022-03-10T01:59:20.869 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.924 INFO:teuthology.orchestra.run.smithi007.stdout:128849018884 2022-03-10T01:59:20.946 INFO:teuthology.orchestra.run.smithi007.stdout:90194313225 2022-03-10T01:59:20.947 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018884 got 128849018884 for osd.11 2022-03-10T01:59:20.948 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.964 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313225 got 90194313225 for osd.3 2022-03-10T01:59:20.964 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:20.973 INFO:teuthology.orchestra.run.smithi007.stdout:128849018883 2022-03-10T01:59:20.992 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018884 got 128849018883 for osd.10 2022-03-10T01:59:21.042 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.9 2022-03-10T01:59:21.608 INFO:teuthology.orchestra.run.smithi007.stdout:115964116998 2022-03-10T01:59:21.627 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116998 for osd.9 2022-03-10T01:59:21.627 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:21.832 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.5 2022-03-10T01:59:21.993 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph osd last-stat-seq osd.10 2022-03-10T01:59:22.434 INFO:teuthology.orchestra.run.smithi007.stdout:115964116996 2022-03-10T01:59:22.458 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116996 got 115964116996 for osd.5 2022-03-10T01:59:22.459 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:22.553 INFO:teuthology.orchestra.run.smithi007.stdout:128849018884 2022-03-10T01:59:22.576 INFO:tasks.rook.ceph_manager.ceph:need seq 128849018884 got 128849018884 for osd.10 2022-03-10T01:59:22.576 DEBUG:teuthology.parallel:result is None 2022-03-10T01:59:22.577 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-10T01:59:22.577 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph pg dump --format=json 2022-03-10T01:59:23.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2022-03-10T01:59:23.118 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2022-03-10T01:59:21.598350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":405516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":66344,"kb_used_data":4960,"kb_used_omap":0,"kb_used_meta":60160,"kb_avail":1124875480,"statfs":{"total":1151940427776,"available":1151872491520,"internally_reserved":0,"allocated":5079040,"data_stored":2892999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61603840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":44,"num_write_kb":1044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":405516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.358934"},"pg_stats":[{"pgid":"1.0","version":"32'125","reported_seq":77,"reported_epoch":32,"state":"active+clean","last_fresh":"2022-03-10T01:59:17.719883+0000","last_change":"2022-03-10T01:59:13.818019+0000","last_active":"2022-03-10T01:59:17.719883+0000","last_peered":"2022-03-10T01:59:17.719883+0000","last_clean":"2022-03-10T01:59:17.719883+0000","last_became_active":"2022-03-10T01:59:13.556024+0000","last_became_peered":"2022-03-10T01:59:13.556024+0000","last_unstale":"2022-03-10T01:59:17.719883+0000","last_undegraded":"2022-03-10T01:59:17.719883+0000","last_fullsized":"2022-03-10T01:59:17.719883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-10T01:59:01.288273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-10T01:59:01.288273+0000","last_clean_scrub_stamp":"2022-03-10T01:59:01.288273+0000","objects_scrubbed":0,"log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-11T09:10:38.889357+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":405516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":405516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2619012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":3,"acting":3,"num_store_stats":6}],"osd_stats":[{"osd":11,"up_from":30,"seq":128849018884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6200,"kb_used_data":760,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738952,"statfs":{"total":95995035648,"available":95988686848,"internally_reserved":0,"allocated":778240,"data_stored":592393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.577}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]}]},{"osd":10,"up_from":30,"seq":128849018884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5108,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740044,"statfs":{"total":95995035648,"available":95989805056,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]}]},{"osd":8,"up_from":30,"seq":128849018884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5108,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740044,"statfs":{"total":95995035648,"available":95989805056,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.571}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.879}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7749999999999999}]}]},{"osd":9,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5168,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":303104,"data_stored":123240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7250000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1379999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4289999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.044}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8980000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6819999999999999}]}]},{"osd":7,"up_from":27,"seq":115964116997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6260,"kb_used_data":756,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93738892,"statfs":{"total":95995035648,"available":95988625408,"internally_reserved":0,"allocated":774144,"data_stored":590728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.627}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]}]},{"osd":6,"up_from":27,"seq":115964116997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6260,"kb_used_data":756,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93738892,"statfs":{"total":95995035648,"available":95988625408,"internally_reserved":0,"allocated":774144,"data_stored":590728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4620000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.762}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.794}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5510000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7109999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.347}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5030000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2490000000000001}]}]},{"osd":1,"up_from":19,"seq":81604378631,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5556,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739596,"statfs":{"total":95995035648,"available":95989346304,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8879999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.351}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4769999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5489999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3769999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9650000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5169999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5556,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739596,"statfs":{"total":95995035648,"available":95989346304,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9910000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.829}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0979999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1859999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.863}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2029999999999998}]}]},{"osd":2,"up_from":20,"seq":85899345927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5552,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739600,"statfs":{"total":95995035648,"available":95989350400,"internally_reserved":0,"allocated":303104,"data_stored":123240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.577}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9239999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9610000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0310000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1760000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0939999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3500000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5236,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739916,"statfs":{"total":95995035648,"available":95989673984,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3500000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1960000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3159999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]}]},{"osd":4,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5172,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739980,"statfs":{"total":95995035648,"available":95989739520,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4910000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.609}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.786}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5619999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8839999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8109999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.859}]}]},{"osd":5,"up_from":27,"seq":115964116996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5168,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":303104,"data_stored":123240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7010000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7509999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":467488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-10T01:59:23.120 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph pg dump --format=json 2022-03-10T01:59:23.606 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:23.607 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2022-03-10T01:59:23.622 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2022-03-10T01:59:23.598940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":405516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":65908,"kb_used_data":5140,"kb_used_omap":0,"kb_used_meta":60288,"kb_avail":1124875916,"statfs":{"total":1151940427776,"available":1151872937984,"internally_reserved":0,"allocated":5263360,"data_stored":3077319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61734912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":27,"num_write":58,"num_write_kb":1306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":405516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.359524"},"pg_stats":[{"pgid":"1.0","version":"32'137","reported_seq":92,"reported_epoch":32,"state":"active+clean","last_fresh":"2022-03-10T01:59:21.118059+0000","last_change":"2022-03-10T01:59:13.818019+0000","last_active":"2022-03-10T01:59:21.118059+0000","last_peered":"2022-03-10T01:59:21.118059+0000","last_clean":"2022-03-10T01:59:21.118059+0000","last_became_active":"2022-03-10T01:59:13.556024+0000","last_became_peered":"2022-03-10T01:59:13.556024+0000","last_unstale":"2022-03-10T01:59:21.118059+0000","last_undegraded":"2022-03-10T01:59:21.118059+0000","last_fullsized":"2022-03-10T01:59:21.118059+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-10T01:59:01.288273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-10T01:59:01.288273+0000","last_clean_scrub_stamp":"2022-03-10T01:59:01.288273+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-11T09:10:38.889357+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":405516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":405516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2826240,"data_stored":2803332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":6}],"osd_stats":[{"osd":11,"up_from":30,"seq":128849018885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6076,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739076,"statfs":{"total":95995035648,"available":95988813824,"internally_reserved":0,"allocated":839680,"data_stored":653833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.577}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]}]},{"osd":10,"up_from":30,"seq":128849018885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5108,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740044,"statfs":{"total":95995035648,"available":95989805056,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6719999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]}]},{"osd":8,"up_from":30,"seq":128849018885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5108,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740044,"statfs":{"total":95995035648,"available":95989805056,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9289999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9580000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7519999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.847}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.206}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5619999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1269999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.2480000000000002}]}]},{"osd":9,"up_from":27,"seq":115964116999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5168,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":303104,"data_stored":123240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.393}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7290000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.833}]}]},{"osd":7,"up_from":27,"seq":115964116998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6136,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739016,"statfs":{"total":95995035648,"available":95988752384,"internally_reserved":0,"allocated":835584,"data_stored":652168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0920000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8819999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3330000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2269999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.204}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2629999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.849}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1219999999999999}]}]},{"osd":6,"up_from":27,"seq":115964116998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6072,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739080,"statfs":{"total":95995035648,"available":95988817920,"internally_reserved":0,"allocated":835584,"data_stored":652168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5529999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7709999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.861}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0710000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.79}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.359}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4089999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8740000000000001}]}]},{"osd":1,"up_from":19,"seq":81604378632,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5556,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739596,"statfs":{"total":95995035648,"available":95989346304,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8879999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.351}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4769999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5489999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3769999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9650000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2919999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5169999999999999}]}]},{"osd":0,"up_from":17,"seq":73014444041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5556,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739596,"statfs":{"total":95995035648,"available":95989346304,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3010000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.462}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.724}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.359}]}]},{"osd":2,"up_from":20,"seq":85899345928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5552,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739600,"statfs":{"total":95995035648,"available":95989350400,"internally_reserved":0,"allocated":303104,"data_stored":123240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.577}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9239999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9610000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0310000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1760000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0939999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3500000000000001}]}]},{"osd":3,"up_from":21,"seq":90194313226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5236,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739916,"statfs":{"total":95995035648,"available":95989673984,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3500000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1960000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3159999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]}]},{"osd":4,"up_from":27,"seq":115964116999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5172,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739980,"statfs":{"total":95995035648,"available":95989739520,"internally_reserved":0,"allocated":307200,"data_stored":124905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.02}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]}]},{"osd":5,"up_from":27,"seq":115964116997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5168,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739984,"statfs":{"total":95995035648,"available":95989743616,"internally_reserved":0,"allocated":303104,"data_stored":123240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4750000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9710000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":405516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-10T01:59:23.623 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-10T01:59:23.624 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-10T01:59:23.624 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-10T01:59:23.625 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph health --format=json 2022-03-10T01:59:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:24.229 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-10T01:59:24.250 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-10T01:59:24.250 INFO:tasks.rook:Rook complete, yielding 2022-03-10T01:59:24.251 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T01:59:24.267 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph -s 2022-03-10T01:59:24.924 INFO:teuthology.orchestra.run.smithi007.stdout: cluster: 2022-03-10T01:59:24.925 INFO:teuthology.orchestra.run.smithi007.stdout: id: 699947c3-1b9d-45f5-8d1c-87f65be1d025 2022-03-10T01:59:24.925 INFO:teuthology.orchestra.run.smithi007.stdout: health: HEALTH_OK 2022-03-10T01:59:24.925 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:24.925 INFO:teuthology.orchestra.run.smithi007.stdout: services: 2022-03-10T01:59:24.926 INFO:teuthology.orchestra.run.smithi007.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-10T01:59:24.926 INFO:teuthology.orchestra.run.smithi007.stdout: mgr: a(active, since 2m) 2022-03-10T01:59:24.926 INFO:teuthology.orchestra.run.smithi007.stdout: osd: 12 osds: 12 up (since 14s), 12 in (since 63s) 2022-03-10T01:59:24.926 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:24.927 INFO:teuthology.orchestra.run.smithi007.stdout: data: 2022-03-10T01:59:24.927 INFO:teuthology.orchestra.run.smithi007.stdout: pools: 1 pools, 1 pgs 2022-03-10T01:59:24.927 INFO:teuthology.orchestra.run.smithi007.stdout: objects: 2 objects, 577 KiB 2022-03-10T01:59:24.927 INFO:teuthology.orchestra.run.smithi007.stdout: usage: 64 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-10T01:59:24.928 INFO:teuthology.orchestra.run.smithi007.stdout: pgs: 1 active+clean 2022-03-10T01:59:24.928 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:24.929 INFO:teuthology.orchestra.run.smithi007.stdout: io: 2022-03-10T01:59:24.929 INFO:teuthology.orchestra.run.smithi007.stdout: client: 2.6 KiB/s rd, 126 KiB/s wr, 2 op/s rd, 5 op/s wr 2022-03-10T01:59:24.929 INFO:teuthology.orchestra.run.smithi007.stdout: recovery: 38 KiB/s, 0 objects/s 2022-03-10T01:59:24.929 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T01:59:24.947 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status 2022-03-10T01:59:25.631 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: rook 2022-03-10T01:59:25.632 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2022-03-10T01:59:25.655 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ps 2022-03-10T01:59:26.247 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-10T01:59:26.247 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi007 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-10T01:59:26.247 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi071 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-10T01:59:26.248 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi184 running (51s) 0s ago 52s - - 4b871cde5e1a 2022-03-10T01:59:26.248 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.a smithi184 running (2m) 0s ago 2m - - 4b871cde5e1a 2022-03-10T01:59:26.248 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi071 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-10T01:59:26.249 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi184 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-10T01:59:26.249 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T01:59:26.250 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi071 running (64s) 0s ago 64s - - 4b871cde5e1a 2022-03-10T01:59:26.250 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi071 running (64s) 0s ago 64s - - 4b871cde5e1a 2022-03-10T01:59:26.250 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 smithi184 running (48s) 0s ago 48s - - 4b871cde5e1a 2022-03-10T01:59:26.250 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 smithi184 running (46s) 0s ago 47s - - 4b871cde5e1a 2022-03-10T01:59:26.251 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi071 running (63s) 0s ago 63s - - 4b871cde5e1a 2022-03-10T01:59:26.251 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi071 running (59s) 0s ago 61s - - 4b871cde5e1a 2022-03-10T01:59:26.251 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi007 running (62s) 0s ago 62s - - 4b871cde5e1a 2022-03-10T01:59:26.251 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi007 running (57s) 0s ago 58s - - 4b871cde5e1a 2022-03-10T01:59:26.251 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi007 running (56s) 0s ago 57s - - 4b871cde5e1a 2022-03-10T01:59:26.252 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi007 running (54s) 0s ago 54s - - 4b871cde5e1a 2022-03-10T01:59:26.252 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 smithi184 running (50s) 0s ago 50s - - 4b871cde5e1a 2022-03-10T01:59:26.252 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 smithi184 running (52s) 0s ago 52s - - 4b871cde5e1a 2022-03-10T01:59:26.270 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls 2022-03-10T01:59:27.035 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-10T01:59:27.036 INFO:teuthology.orchestra.run.smithi007.stdout:crash 3/3 0s ago 2m * 2022-03-10T01:59:27.036 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 1/1 0s ago 2m count:1 2022-03-10T01:59:27.036 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 0s ago 5m count:3 2022-03-10T01:59:27.037 INFO:teuthology.orchestra.run.smithi007.stdout:osd 24 0s ago 65s 2022-03-10T01:59:27.037 INFO:teuthology.orchestra.run.smithi007.stdout:osd.all-available-devices 0 0s ago - * 2022-03-10T01:59:27.055 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch device ls 2022-03-10T01:59:27.576 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-10T01:59:27.577 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T01:59:27.577 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T01:59:27.577 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T01:59:27.577 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T01:59:27.578 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T01:59:27.578 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T01:59:27.579 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T01:59:27.579 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T01:59:27.579 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T01:59:27.579 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T01:59:27.579 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T01:59:27.580 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T01:59:27.595 INFO:teuthology.run_tasks:Running task exec... 2022-03-10T01:59:27.608 INFO:teuthology.task.exec:Executing custom commands... 2022-03-10T01:59:27.608 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi007.front.sepia.ceph.com 2022-03-10T01:59:27.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-10T01:59:27.609 DEBUG:teuthology.orchestra.run.smithi007:> toolbox() { 2022-03-10T01:59:27.609 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-10T01:59:27.610 DEBUG:teuthology.orchestra.run.smithi007:> } 2022-03-10T01:59:27.610 DEBUG:teuthology.orchestra.run.smithi007:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-10T01:59:27.610 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch osd rm 0 --force 2022-03-10T01:59:27.611 DEBUG:teuthology.orchestra.run.smithi007:> removed_pv="" 2022-03-10T01:59:27.612 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$removed_pv" = "" ] 2022-03-10T01:59:27.612 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-10T01:59:27.612 DEBUG:teuthology.orchestra.run.smithi007:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-10T01:59:27.613 DEBUG:teuthology.orchestra.run.smithi007:> sleep 3s 2022-03-10T01:59:27.613 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-10T01:59:27.613 DEBUG:teuthology.orchestra.run.smithi007:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-10T01:59:27.613 DEBUG:teuthology.orchestra.run.smithi007:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-10T01:59:27.614 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch device zap $host $target_path --force 2022-03-10T01:59:27.614 DEBUG:teuthology.orchestra.run.smithi007:> zap_completion="0" 2022-03-10T01:59:27.614 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$zap_completion" = "0" ] 2022-03-10T01:59:27.615 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-10T01:59:27.615 DEBUG:teuthology.orchestra.run.smithi007:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-10T01:59:27.615 DEBUG:teuthology.orchestra.run.smithi007:> sleep 3s 2022-03-10T01:59:27.616 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-10T01:59:27.616 DEBUG:teuthology.orchestra.run.smithi007:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-10T01:59:27.616 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch apply osd --all-available-devices 2022-03-10T01:59:27.616 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-10T01:59:27.617 DEBUG:teuthology.orchestra.run.smithi007:> num_osd="0" 2022-03-10T01:59:27.617 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-10T01:59:27.617 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-10T01:59:27.618 DEBUG:teuthology.orchestra.run.smithi007:> echo "waiting for osd to come back up" 2022-03-10T01:59:27.618 DEBUG:teuthology.orchestra.run.smithi007:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-10T01:59:27.618 DEBUG:teuthology.orchestra.run.smithi007:> sleep 30s 2022-03-10T01:59:27.619 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-10T01:59:27.619 DEBUG:teuthology.orchestra.run.smithi007:> ' 2022-03-10T01:59:27.635 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-10T01:59:27.636 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:59:27.636 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-10T01:59:30.519 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:59:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:+ orig_num_osd=12 2022-03-10T01:59:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-10T01:59:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-10T01:59:31.229 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:59:31.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:31.843343 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T01:59:31.893 INFO:tasks.rook.operator.smithi007.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T01:59:31.893 INFO:tasks.rook.operator.smithi007.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T01:59:31.894 INFO:tasks.rook.operator.smithi007.stdout: ... // 3 identical fields 2022-03-10T01:59:31.894 INFO:tasks.rook.operator.smithi007.stdout: Config: nil, 2022-03-10T01:59:31.894 INFO:tasks.rook.operator.smithi007.stdout: Selection: {}, 2022-03-10T01:59:31.894 INFO:tasks.rook.operator.smithi007.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T01:59:31.895 INFO:tasks.rook.operator.smithi007.stdout: ... // 2 identical elements 2022-03-10T01:59:31.895 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi007-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi007-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:59:31.895 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi007-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi007-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:59:31.895 INFO:tasks.rook.operator.smithi007.stdout:- { 2022-03-10T01:59:31.896 INFO:tasks.rook.operator.smithi007.stdout:- Name: "smithi071-lv-1", 2022-03-10T01:59:31.896 INFO:tasks.rook.operator.smithi007.stdout:- Count: 1, 2022-03-10T01:59:31.896 INFO:tasks.rook.operator.smithi007.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-10T01:59:31.896 INFO:tasks.rook.operator.smithi007.stdout:- }, 2022-03-10T01:59:31.896 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi071-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi071-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:59:31.897 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi071-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi071-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:59:31.897 INFO:tasks.rook.operator.smithi007.stdout: ... // 5 identical elements 2022-03-10T01:59:31.897 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-10T01:59:31.897 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-10T01:59:31.898 INFO:tasks.rook.operator.smithi007.stdout: Annotations: nil, 2022-03-10T01:59:31.898 INFO:tasks.rook.operator.smithi007.stdout: Labels: nil, 2022-03-10T01:59:31.898 INFO:tasks.rook.operator.smithi007.stdout: ... // 20 identical fields 2022-03-10T01:59:31.898 INFO:tasks.rook.operator.smithi007.stdout: } 2022-03-10T01:59:31.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:31.843512 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T01:59:31.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:31.843892 I | operator: successfully started the controller-runtime manager 2022-03-10T01:59:31.899 INFO:tasks.rook.operator.smithi007.stdout:I0310 01:59:31.843962 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T01:59:31.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:31.874291 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T01:59:31.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:31.874317 I | operator: watching all namespaces for Ceph CRs 2022-03-10T01:59:31.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:31.874351 I | operator: setting up schemes 2022-03-10T01:59:31.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:31.878393 I | operator: setting up the controller-runtime manager 2022-03-10T01:59:32.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.082219 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T01:59:32.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166784 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T01:59:32.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166819 I | ceph-cluster-controller: successfully started 2022-03-10T01:59:32.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166864 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T01:59:32.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166879 I | ceph-crashcollector-controller: successfully started 2022-03-10T01:59:32.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166892 I | ceph-block-pool-controller: successfully started 2022-03-10T01:59:32.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166908 I | ceph-object-store-user-controller: successfully started 2022-03-10T01:59:32.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166923 I | ceph-object-realm-controller: successfully started 2022-03-10T01:59:32.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166937 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T01:59:32.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.166946 I | ceph-object-zone-controller: successfully started 2022-03-10T01:59:32.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167075 I | ceph-object-controller: successfully started 2022-03-10T01:59:32.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167102 I | ceph-file-controller: successfully started 2022-03-10T01:59:32.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167125 I | ceph-nfs-controller: successfully started 2022-03-10T01:59:32.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167146 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T01:59:32.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167166 I | ceph-client-controller: successfully started 2022-03-10T01:59:32.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167187 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T01:59:32.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167212 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T01:59:32.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167227 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T01:59:32.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167246 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T01:59:32.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167257 I | ceph-bucket-topic: successfully started 2022-03-10T01:59:32.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167267 I | ceph-bucket-notification: successfully started 2022-03-10T01:59:32.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167285 I | ceph-bucket-notification: successfully started 2022-03-10T01:59:32.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.167295 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T01:59:32.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.170355 I | operator: starting the controller-runtime manager 2022-03-10T01:59:32.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.274512 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T01:59:32.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.274596 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T01:59:32.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.274617 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T01:59:32.277 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.275088 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:59:32.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.371149 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T01:59:32.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.392999 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:59:32.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.393070 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T01:59:32.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.393091 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:59:32.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.393099 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:59:32.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.395354 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T01:59:32.396 INFO:tasks.rook.operator.smithi007.stdout:I0310 01:59:32.395451 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:59:32.434 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.436260 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T01:59:32.446 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.448746 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:59:32.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544511 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T01:59:32.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544529 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T01:59:32.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544533 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T01:59:32.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544536 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T01:59:32.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544540 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T01:59:32.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544543 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T01:59:32.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544546 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T01:59:32.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544550 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T01:59:32.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544557 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T01:59:32.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544563 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T01:59:32.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544566 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T01:59:32.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544570 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T01:59:32.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544573 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T01:59:32.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544575 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T01:59:32.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544579 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T01:59:32.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544584 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T01:59:32.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544632 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:59:32.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:32.544646 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:59:34.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:34.713656 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-10T01:59:34.801 INFO:teuthology.orchestra.run.smithi007.stdout:Success 2022-03-10T01:59:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:+ removed_pv= 2022-03-10T01:59:34.846 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' '' = '' ']' 2022-03-10T01:59:34.847 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get pv 2022-03-10T01:59:34.847 INFO:teuthology.orchestra.run.smithi007.stderr:++ grep Released 2022-03-10T01:59:34.847 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f1 -d ' ' 2022-03-10T01:59:34.912 INFO:teuthology.orchestra.run.smithi007.stderr:+ removed_pv=smithi071-lv-1 2022-03-10T01:59:34.912 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 3s 2022-03-10T01:59:37.913 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' smithi071-lv-1 = '' ']' 2022-03-10T01:59:37.914 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get pv smithi071-lv-1 -o 'jsonpath={.spec.local.path}' 2022-03-10T01:59:38.154 INFO:teuthology.orchestra.run.smithi007.stderr:+ target_path=/dev/vg_nvme/lv_1 2022-03-10T01:59:38.155 INFO:teuthology.orchestra.run.smithi007.stderr:++ echo smithi071-lv-1 2022-03-10T01:59:38.155 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f1 -d - 2022-03-10T01:59:38.157 INFO:teuthology.orchestra.run.smithi007.stderr:+ host=smithi071 2022-03-10T01:59:38.157 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch device zap smithi071 /dev/vg_nvme/lv_1 --force 2022-03-10T01:59:38.157 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi071 /dev/vg_nvme/lv_1 --force 2022-03-10T01:59:38.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469467 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T01:59:38.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469492 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T01:59:38.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469497 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:59:38.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469500 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:59:38.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469503 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:59:38.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469505 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:59:38.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469508 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:59:38.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469510 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:59:38.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469513 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T01:59:38.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469515 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T01:59:38.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469518 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:59:38.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469521 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:59:38.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469524 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:59:38.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469526 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:59:38.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469529 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T01:59:38.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469531 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T01:59:38.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469534 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T01:59:38.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469536 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T01:59:38.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469547 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T01:59:38.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469554 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:59:38.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469558 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:59:38.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469563 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T01:59:38.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469567 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T01:59:38.475 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469573 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T01:59:38.475 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.469578 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T01:59:38.475 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:59:38.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.682757 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T01:59:38.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.688157 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:59:38.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.688183 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:59:38.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.688188 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:59:38.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.688190 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:59:38.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.688193 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:59:38.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.688195 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:59:38.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.688198 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T01:59:38.834 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.835744 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:59:38.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.835771 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:59:38.836 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.835778 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:59:38.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:38.890788 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T01:59:39.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.108912 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:59:39.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.108945 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:59:39.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.108954 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T01:59:39.187 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_1 on smithi071 zapped 2022-03-10T01:59:39.212 INFO:teuthology.orchestra.run.smithi007.stderr:+ zap_completion=0 2022-03-10T01:59:39.213 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 0 = 0 ']' 2022-03-10T01:59:39.213 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-10T01:59:39.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.265909 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:59:39.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.265943 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:59:39.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.265956 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:59:39.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.486183 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T01:59:39.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.486225 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T01:59:39.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.486247 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T01:59:39.487 INFO:teuthology.orchestra.run.smithi007.stderr:+ zap_completion= 2022-03-10T01:59:39.487 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 3s 2022-03-10T01:59:39.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:39.861587 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:59:40.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.034501 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T01:59:40.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.058628 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:59:40.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.058918 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:59:40.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.256117 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:59:40.261 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.262180 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T01:59:40.261 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.262204 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:59:40.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.266801 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T01:59:40.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.485479 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T01:59:40.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:40.801873 I | op-mon: start running mons 2022-03-10T01:59:41.216 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:41.217133 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:59:41.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:41.540879 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789","10.101.240.174:6789","10.111.0.148:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-10T01:59:41.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:41.548808 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T01:59:41.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:41.549068 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T01:59:42.487 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' '' = 0 ']' 2022-03-10T01:59:42.488 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl patch pv smithi071-lv-1 -p '{"spec":{"claimRef": null}}' 2022-03-10T01:59:42.557 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi071-lv-1 patched 2022-03-10T01:59:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-10T01:59:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-10T01:59:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:59:42.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:42.896992 I | op-mon: targeting the mon count 3 2022-03-10T01:59:42.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:42.905496 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:59:43.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.259496 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T01:59:43.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.259523 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:59:43.325 INFO:teuthology.orchestra.run.smithi007.stdout:['Success'] 2022-03-10T01:59:43.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.330566 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-10T01:59:43.331 INFO:tasks.rook.operator.smithi007.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", AllowUnsupported: true}, 2022-03-10T01:59:43.331 INFO:tasks.rook.operator.smithi007.stdout: Storage: v1.StorageScopeSpec{ 2022-03-10T01:59:43.332 INFO:tasks.rook.operator.smithi007.stdout: ... // 3 identical fields 2022-03-10T01:59:43.332 INFO:tasks.rook.operator.smithi007.stdout: Config: nil, 2022-03-10T01:59:43.332 INFO:tasks.rook.operator.smithi007.stdout: Selection: {}, 2022-03-10T01:59:43.332 INFO:tasks.rook.operator.smithi007.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-10T01:59:43.333 INFO:tasks.rook.operator.smithi007.stdout: ... // 9 identical elements 2022-03-10T01:59:43.333 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi184-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi184-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:59:43.333 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi184-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi184-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-10T01:59:43.333 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-10T01:59:43.334 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi071-lv-1", 2022-03-10T01:59:43.334 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-10T01:59:43.334 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-10T01:59:43.334 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-10T01:59:43.335 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-10T01:59:43.335 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-10T01:59:43.335 INFO:tasks.rook.operator.smithi007.stdout: Annotations: nil, 2022-03-10T01:59:43.335 INFO:tasks.rook.operator.smithi007.stdout: Labels: nil, 2022-03-10T01:59:43.335 INFO:tasks.rook.operator.smithi007.stdout: ... // 20 identical fields 2022-03-10T01:59:43.336 INFO:tasks.rook.operator.smithi007.stdout: } 2022-03-10T01:59:43.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.330819 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T01:59:43.336 INFO:tasks.rook.operator.smithi007.stdout:I0310 01:59:43.331206 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T01:59:43.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.347813 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T01:59:43.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.347831 I | operator: watching all namespaces for Ceph CRs 2022-03-10T01:59:43.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.347849 I | operator: setting up schemes 2022-03-10T01:59:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-10T01:59:43.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.350036 I | operator: setting up the controller-runtime manager 2022-03-10T01:59:43.410 INFO:teuthology.orchestra.run.smithi007.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-10T01:59:43.416 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=0 2022-03-10T01:59:43.416 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 0 '!=' 12 ']' 2022-03-10T01:59:43.416 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T01:59:43.417 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-10T01:59:43.419 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-10T01:59:43.419 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T01:59:43.419 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-10T01:59:43.486 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T01:59:43.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.552524 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T01:59:43.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556585 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T01:59:43.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556611 I | ceph-cluster-controller: successfully started 2022-03-10T01:59:43.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556646 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T01:59:43.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556664 I | ceph-crashcollector-controller: successfully started 2022-03-10T01:59:43.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556682 I | ceph-block-pool-controller: successfully started 2022-03-10T01:59:43.558 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556697 I | ceph-object-store-user-controller: successfully started 2022-03-10T01:59:43.558 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556709 I | ceph-object-realm-controller: successfully started 2022-03-10T01:59:43.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556719 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T01:59:43.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556730 I | ceph-object-zone-controller: successfully started 2022-03-10T01:59:43.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556833 I | ceph-object-controller: successfully started 2022-03-10T01:59:43.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556857 I | ceph-file-controller: successfully started 2022-03-10T01:59:43.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556885 I | ceph-nfs-controller: successfully started 2022-03-10T01:59:43.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556903 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T01:59:43.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556920 I | ceph-client-controller: successfully started 2022-03-10T01:59:43.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556934 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T01:59:43.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556959 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T01:59:43.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556974 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T01:59:43.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556984 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T01:59:43.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.556995 I | ceph-bucket-topic: successfully started 2022-03-10T01:59:43.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.557005 I | ceph-bucket-notification: successfully started 2022-03-10T01:59:43.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.557020 I | ceph-bucket-notification: successfully started 2022-03-10T01:59:43.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.557033 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T01:59:43.566 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.558031 I | operator: starting the controller-runtime manager 2022-03-10T01:59:43.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.664883 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T01:59:43.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.664937 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T01:59:43.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.664957 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T01:59:43.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.665840 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:59:43.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.673070 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T01:59:43.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.796117 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T01:59:43.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.796140 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T01:59:43.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.796194 W | op-mon: failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. failed to apply default Ceph configurations: failed to set one or more Ceph configs: failed to set ceph config in the centralized mon configuration database; you may need to use the rook-config-override ConfigMap. output: : context canceled 2022-03-10T01:59:43.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.796200 I | op-mon: checking for basic quorum with existing mons 2022-03-10T01:59:43.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.814256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-10T01:59:43.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.814495 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T01:59:43.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.923539 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:59:43.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.923616 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-10T01:59:43.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.923629 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:59:43.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:43.925714 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-10T01:59:43.925 INFO:tasks.rook.operator.smithi007.stdout:I0310 01:59:43.925819 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-10T01:59:44.012 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-10T01:59:44.012 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-10T01:59:44.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.316262 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-10T01:59:44.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.696462 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:59:44.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.714998 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-10 01:59:44.706270802 +0000 UTC m=+445.833769900 LastTransitionTime:2022-03-10 01:59:44.706270638 +0000 UTC m=+445.833769768}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:59:44.715 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.715015 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:59:44.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895285 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-10T01:59:44.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895312 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-10T01:59:44.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895320 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-10T01:59:44.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895327 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-10T01:59:44.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895335 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-10T01:59:44.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895343 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-10T01:59:44.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895351 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-10T01:59:44.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895359 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-10T01:59:44.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895367 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-10T01:59:44.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895374 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-10T01:59:44.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895382 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-10T01:59:44.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895390 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-10T01:59:44.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895398 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-10T01:59:44.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895404 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-10T01:59:44.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895411 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-10T01:59:44.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895419 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-10T01:59:44.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895479 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:59:44.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:44.895490 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:59:45.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:45.106395 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T01:59:45.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:45.126173 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-10 01:59:45.117006303 +0000 UTC m=+446.244505422 LastTransitionTime:2022-03-10 01:59:45.117006103 +0000 UTC m=+446.244505242}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T01:59:45.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:45.126194 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T01:59:45.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:45.699938 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-10T01:59:45.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:45.853680 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T01:59:48.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:48.862943 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T01:59:51.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:51.924486 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-10T01:59:52.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432579 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-10T01:59:52.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432621 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-10T01:59:52.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432635 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:59:52.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432647 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-10T01:59:52.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432661 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:59:52.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432670 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-10T01:59:52.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432678 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:59:52.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432685 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-10T01:59:52.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432693 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T01:59:52.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432703 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-10T01:59:52.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432715 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:59:52.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432727 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-10T01:59:52.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432738 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:59:52.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432749 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-10T01:59:52.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432761 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-10T01:59:52.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432775 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-10T01:59:52.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432788 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-10T01:59:52.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432799 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-10T01:59:52.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432811 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-10T01:59:52.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432823 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:59:52.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432835 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-10T01:59:52.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432849 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-10T01:59:52.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432861 I | ceph-csi: Kubernetes version is 1.23 2022-03-10T01:59:52.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432881 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-10T01:59:52.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.432894 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-10T01:59:52.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.476931 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-10T01:59:52.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.486247 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:59:52.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.486268 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:59:52.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.486273 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:59:52.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.486277 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:59:52.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.486281 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:59:52.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.486286 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:59:52.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.486290 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-10T01:59:52.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.632300 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:59:52.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.632324 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:59:52.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.632330 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:59:52.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.677769 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-10T01:59:52.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.851063 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-10T01:59:52.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.851081 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-10T01:59:52.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.851085 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-10T01:59:52.922 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.922345 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-10T01:59:52.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.922377 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-10T01:59:52.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.922385 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-10T01:59:52.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:52.974427 I | ceph-csi: successfully started CSI CephFS driver 2022-03-10T01:59:53.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:53.171981 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:59:53.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:53.177060 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-10T01:59:53.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:53.177076 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-10T01:59:53.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:53.181496 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-10T01:59:54.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 01:59:54.931033 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-10T02:00:02.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:02.533608 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T02:00:02.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:02.533644 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T02:00:02.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:02.533653 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T02:00:02.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:02.533678 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-10T02:00:02.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:02.533693 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T02:00:02.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:02.533768 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11025 quincy", pursuing 2022-03-10T02:00:02.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:02.818285 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:00:02.822 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:02.818408 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:00:03.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.039647 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T02:00:03.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.039775 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T02:00:03.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.039977 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T02:00:03.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.040535 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T02:00:03.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.485810 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T02:00:03.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.495813 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11025 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3" 2022-03-10T02:00:03.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.516445 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-10 02:00:03.507893845 +0000 UTC m=+464.635392935 LastTransitionTime:2022-03-10 02:00:03.507893723 +0000 UTC m=+464.635392833}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:03.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.535022 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-10 02:00:03.516083721 +0000 UTC m=+464.643582847 LastTransitionTime:2022-03-10 02:00:03.516083531 +0000 UTC m=+464.643582674}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:03.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.550551 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-03-10 02:00:03.53166015 +0000 UTC m=+464.659159276 LastTransitionTime:2022-03-10 02:00:03.53165996 +0000 UTC m=+464.659159096}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:03.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.550577 I | op-mon: start running mons 2022-03-10T02:00:03.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.564839 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:00:03.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:03.566085 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mons LastHeartbeatTime:2022-03-10 02:00:03.548147051 +0000 UTC m=+464.675646156 LastTransitionTime:2022-03-10 02:00:03.548146906 +0000 UTC m=+464.675646025}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:04.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:04.205520 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789","10.101.240.174:6789","10.111.0.148:6789"]}] data:b=10.101.240.174:6789,c=10.111.0.148:6789,a=10.99.228.129:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-10T02:00:04.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:04.799037 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T02:00:04.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:04.799320 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T02:00:06.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:06.400135 I | op-mon: targeting the mon count 3 2022-03-10T02:00:06.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:06.408718 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T02:00:06.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:06.752488 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T02:00:06.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:06.752518 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T02:00:07.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:07.121248 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T02:00:07.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:07.121268 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T02:00:07.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:07.479503 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T02:00:07.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:07.479528 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T02:00:07.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:07.822345 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T02:00:07.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:07.822382 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T02:00:08.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:08.152903 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T02:00:08.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:08.152932 I | op-mon: checking for basic quorum with existing mons 2022-03-10T02:00:08.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:08.280827 I | op-mon: mon "b" endpoint is [v2:10.101.240.174:3300,v1:10.101.240.174:6789] 2022-03-10T02:00:08.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:08.456641 I | op-mon: mon "c" endpoint is [v2:10.111.0.148:3300,v1:10.111.0.148:6789] 2022-03-10T02:00:08.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:08.801375 I | op-mon: mon "a" endpoint is [v2:10.99.228.129:3300,v1:10.99.228.129:6789] 2022-03-10T02:00:09.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:09.400075 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.99.228.129:6789","10.101.240.174:6789","10.111.0.148:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-10T02:00:09.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:09.997899 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T02:00:10.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:09.998187 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T02:00:10.419 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:10.417770 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T02:00:10.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:10.426593 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T02:00:10.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:10.426619 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T02:00:11.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:11.008318 I | op-mon: mons running: [b c a] 2022-03-10T02:00:11.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:11.457477 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T02:00:11.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:11.464684 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T02:00:11.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:11.473521 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T02:00:11.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:11.473541 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T02:00:11.610 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:11.608007 I | op-mon: mons running: [b c a] 2022-03-10T02:00:12.011 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:12.010282 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T02:00:12.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:12.035811 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T02:00:12.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:12.047024 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T02:00:12.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:12.047042 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T02:00:12.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:12.208187 I | op-mon: mons running: [b c a] 2022-03-10T02:00:12.635 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:12.633164 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T02:00:12.635 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:12.633193 I | op-mon: mons created: 3 2022-03-10T02:00:13.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:13.050951 I | op-mon: waiting for mon quorum with [b c a] 2022-03-10T02:00:13.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:13.083986 I | op-mon: mons running: [b c a] 2022-03-10T02:00:13.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:13.521908 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T02:00:13.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:13.521969 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T02:00:13.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:13.522040 I | op-mon: start running mons 2022-03-10T02:00:13.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:13.528394 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:00:13.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:13.950488 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T02:00:14.014 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-10T02:00:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T02:00:14.016 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T02:00:14.016 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-10T02:00:14.016 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T02:00:14.017 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-10T02:00:14.086 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T02:00:14.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:14.199129 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.101.240.174:6789","10.111.0.148:6789","10.99.228.129:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-10T02:00:14.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:14.367224 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T02:00:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-10T02:00:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-10T02:00:14.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:14.794579 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T02:00:14.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:14.798636 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T02:00:14.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:14.798806 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T02:00:17.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:17.599245 I | op-mon: targeting the mon count 3 2022-03-10T02:00:17.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:17.609507 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T02:00:17.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:17.958705 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-10T02:00:17.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:17.958740 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T02:00:18.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:17.999962 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T02:00:18.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:17.999991 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T02:00:18.296 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:18.294079 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-10T02:00:18.296 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:18.294121 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T02:00:18.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:18.469529 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T02:00:18.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:18.693562 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-10T02:00:18.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:18.693592 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-10T02:00:18.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:18.909198 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T02:00:18.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:18.909229 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T02:00:18.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:18.909236 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T02:00:19.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.041371 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-10T02:00:19.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.041392 I | op-config: deleting "log file" option from the mon configuration database 2022-03-10T02:00:19.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.351959 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T02:00:19.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.390189 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mgr(s) LastHeartbeatTime:2022-03-10 02:00:19.383557111 +0000 UTC m=+480.511056220 LastTransitionTime:2022-03-10 02:00:19.383556851 +0000 UTC m=+480.511056030}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:19.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.390214 I | op-mgr: start running mgr 2022-03-10T02:00:19.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.390229 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T02:00:19.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.413680 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-10T02:00:19.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.413709 I | op-mon: checking for basic quorum with existing mons 2022-03-10T02:00:19.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:19.599933 I | op-mon: mon "a" endpoint is [v2:10.99.228.129:3300,v1:10.99.228.129:6789] 2022-03-10T02:00:20.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:20.799327 I | op-mon: mon "b" endpoint is [v2:10.101.240.174:3300,v1:10.101.240.174:6789] 2022-03-10T02:00:21.035 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:21.033190 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T02:00:21.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:21.045286 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T02:00:21.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:21.049255 I | op-mgr: setting services to point to mgr "a" 2022-03-10T02:00:22.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:22.401037 I | op-mon: mon "c" endpoint is [v2:10.111.0.148:3300,v1:10.111.0.148:6789] 2022-03-10T02:00:23.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:22.998621 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T02:00:23.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:22.998812 I | op-mgr: successful modules: balancer 2022-03-10T02:00:23.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.010144 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph OSDs LastHeartbeatTime:2022-03-10 02:00:22.998757629 +0000 UTC m=+484.126256742 LastTransitionTime:2022-03-10 02:00:22.998757462 +0000 UTC m=+484.126256575}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:23.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.010172 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T02:00:23.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.010180 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T02:00:23.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.044976 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T02:00:23.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400437 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-1" 2022-03-10T02:00:23.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400485 I | op-osd: OSD PVC "smithi007-lv-1-data-0jzl5l" already exists 2022-03-10T02:00:23.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400523 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-2" 2022-03-10T02:00:23.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400543 I | op-osd: OSD PVC "smithi007-lv-2-data-08blxh" already exists 2022-03-10T02:00:23.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400558 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-3" 2022-03-10T02:00:23.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400576 I | op-osd: OSD PVC "smithi007-lv-3-data-0xslw4" already exists 2022-03-10T02:00:23.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400595 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-4" 2022-03-10T02:00:23.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400619 I | op-osd: OSD PVC "smithi007-lv-4-data-0cfg7m" already exists 2022-03-10T02:00:23.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400637 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi071-lv-2" 2022-03-10T02:00:23.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400658 I | op-osd: OSD PVC "smithi071-lv-2-data-0lmhm6" already exists 2022-03-10T02:00:23.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400681 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi071-lv-3" 2022-03-10T02:00:23.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400693 I | op-osd: OSD PVC "smithi071-lv-3-data-0xjgfz" already exists 2022-03-10T02:00:23.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400701 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi071-lv-4" 2022-03-10T02:00:23.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400719 I | op-osd: OSD PVC "smithi071-lv-4-data-0q5d9z" already exists 2022-03-10T02:00:23.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400732 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi184-lv-1" 2022-03-10T02:00:23.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400752 I | op-osd: OSD PVC "smithi184-lv-1-data-096cgx" already exists 2022-03-10T02:00:23.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400766 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi184-lv-2" 2022-03-10T02:00:23.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400797 I | op-osd: OSD PVC "smithi184-lv-2-data-0l5x8k" already exists 2022-03-10T02:00:23.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400816 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi184-lv-3" 2022-03-10T02:00:23.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400835 I | op-osd: OSD PVC "smithi184-lv-3-data-09sg48" already exists 2022-03-10T02:00:23.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400849 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi184-lv-4" 2022-03-10T02:00:23.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400870 I | op-osd: OSD PVC "smithi184-lv-4-data-0n7czp" already exists 2022-03-10T02:00:23.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.400885 I | op-osd: creating 1 new PVCs for device set "smithi071-lv-1" 2022-03-10T02:00:23.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.599650 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.101.240.174:6789","10.111.0.148:6789","10.99.228.129:6789"]}] data:a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 mapping:{"node":{"a":{"Name":"smithi071","Hostname":"smithi071","Address":"172.21.15.71"},"b":{"Name":"smithi184","Hostname":"smithi184","Address":"172.21.15.184"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-10T02:00:23.804 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.802646 I | op-osd: successfully provisioned PVC "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:23.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838678 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0jzl5l" 2022-03-10T02:00:23.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838703 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-1-data-0jzl5l" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838710 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-08blxh" 2022-03-10T02:00:23.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838715 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-2-data-08blxh" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838721 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0xslw4" 2022-03-10T02:00:23.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838726 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-3-data-0xslw4" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838731 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0cfg7m" 2022-03-10T02:00:23.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838737 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-4-data-0cfg7m" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838743 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-2-data-0lmhm6" 2022-03-10T02:00:23.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838750 I | op-osd: skipping OSD prepare job creation for PVC "smithi071-lv-2-data-0lmhm6" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838758 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-3-data-0xjgfz" 2022-03-10T02:00:23.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838765 I | op-osd: skipping OSD prepare job creation for PVC "smithi071-lv-3-data-0xjgfz" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838771 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-4-data-0q5d9z" 2022-03-10T02:00:23.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838778 I | op-osd: skipping OSD prepare job creation for PVC "smithi071-lv-4-data-0q5d9z" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838784 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-1-data-096cgx" 2022-03-10T02:00:23.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838791 I | op-osd: skipping OSD prepare job creation for PVC "smithi184-lv-1-data-096cgx" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838801 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-2-data-0l5x8k" 2022-03-10T02:00:23.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838807 I | op-osd: skipping OSD prepare job creation for PVC "smithi184-lv-2-data-0l5x8k" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838814 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-3-data-09sg48" 2022-03-10T02:00:23.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838819 I | op-osd: skipping OSD prepare job creation for PVC "smithi184-lv-3-data-09sg48" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838826 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-4-data-0n7czp" 2022-03-10T02:00:23.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838832 I | op-osd: skipping OSD prepare job creation for PVC "smithi184-lv-4-data-0n7czp" because OSD daemon using the PVC already exists 2022-03-10T02:00:23.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:23.838838 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:24.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:24.626809 I | op-osd: started OSD provisioning job for PVC "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:24.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:24.626849 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T02:00:24.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:24.626860 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T02:00:24.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:24.681132 I | op-mgr: successful modules: prometheus 2022-03-10T02:00:24.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:24.682955 I | op-mgr: successful modules: dashboard 2022-03-10T02:00:24.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:24.684152 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T02:00:24.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:24.813366 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-10T02:00:24.818 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:24.813571 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-10T02:00:25.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.022311 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0kfwx5 is "starting" 2022-03-10T02:00:25.127 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.123909 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:25.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.130612 I | op-osd: updating OSD 1 on PVC "smithi071-lv-2-data-0lmhm6" 2022-03-10T02:00:25.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.130637 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-2-data-0lmhm6" 2022-03-10T02:00:25.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.147757 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 1 on PVC "smithi071-lv-2-data-0lmhm6" LastHeartbeatTime:2022-03-10 02:00:25.139419332 +0000 UTC m=+486.266918424 LastTransitionTime:2022-03-10 02:00:25.139419163 +0000 UTC m=+486.266918294}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:25.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.270977 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:25.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.276968 I | op-osd: updating OSD 10 on PVC "smithi184-lv-3-data-09sg48" 2022-03-10T02:00:25.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.276995 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-3-data-09sg48" 2022-03-10T02:00:25.300 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.298015 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 10 on PVC "smithi184-lv-3-data-09sg48" LastHeartbeatTime:2022-03-10 02:00:25.287125998 +0000 UTC m=+486.414625094 LastTransitionTime:2022-03-10 02:00:25.287125824 +0000 UTC m=+486.414624952}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:25.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.422665 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-10T02:00:25.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.434301 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-10T02:00:25.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.434335 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T02:00:25.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.483486 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:25.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.531922 I | op-osd: updating OSD 11 on PVC "smithi184-lv-4-data-0n7czp" 2022-03-10T02:00:25.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.531973 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-4-data-0n7czp" 2022-03-10T02:00:25.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:25.750763 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 11 on PVC "smithi184-lv-4-data-0n7czp" LastHeartbeatTime:2022-03-10 02:00:25.741597863 +0000 UTC m=+486.869096971 LastTransitionTime:2022-03-10 02:00:25.741597623 +0000 UTC m=+486.869096785}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:26.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:26.007165 I | op-mon: mons running: [a b c] 2022-03-10T02:00:26.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:26.235449 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:26.334 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:26.330685 I | op-osd: updating OSD 2 on PVC "smithi071-lv-3-data-0xjgfz" 2022-03-10T02:00:26.334 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:26.330719 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-3-data-0xjgfz" 2022-03-10T02:00:26.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:26.457245 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T02:00:26.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:26.549276 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 2 on PVC "smithi071-lv-3-data-0xjgfz" LastHeartbeatTime:2022-03-10 02:00:26.537214994 +0000 UTC m=+487.664714118 LastTransitionTime:2022-03-10 02:00:26.537214774 +0000 UTC m=+487.664713927}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:26.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:26.640909 I | op-mgr: successful modules: orchestrator modules 2022-03-10T02:00:26.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:26.739987 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-10T02:00:27.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.140012 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-10T02:00:27.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.140039 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T02:00:27.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.175446 I | op-mon: mons running: [a b c] 2022-03-10T02:00:27.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.446677 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:27.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.531466 I | op-osd: updating OSD 3 on PVC "smithi071-lv-4-data-0q5d9z" 2022-03-10T02:00:27.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.531504 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-4-data-0q5d9z" 2022-03-10T02:00:27.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.607604 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T02:00:27.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.749708 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 3 on PVC "smithi071-lv-4-data-0q5d9z" LastHeartbeatTime:2022-03-10 02:00:27.738377574 +0000 UTC m=+488.865876678 LastTransitionTime:2022-03-10 02:00:27.738377296 +0000 UTC m=+488.865876481}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:27.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:27.931182 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-10T02:00:28.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.335959 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-10T02:00:28.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.335987 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T02:00:28.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.371848 I | op-mon: mons running: [a b c] 2022-03-10T02:00:28.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.636860 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:28.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.731348 I | op-osd: updating OSD 4 on PVC "smithi007-lv-2-data-08blxh" 2022-03-10T02:00:28.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.731381 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-08blxh" 2022-03-10T02:00:28.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.799585 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T02:00:28.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.799612 I | op-mon: mons created: 3 2022-03-10T02:00:28.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:28.948301 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 4 on PVC "smithi007-lv-2-data-08blxh" LastHeartbeatTime:2022-03-10 02:00:28.937360241 +0000 UTC m=+490.064859345 LastTransitionTime:2022-03-10 02:00:28.937360057 +0000 UTC m=+490.064859193}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:29.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:29.208602 I | op-mon: waiting for mon quorum with [a b c] 2022-03-10T02:00:29.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:29.296734 I | op-mon: mons running: [a b c] 2022-03-10T02:00:29.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:29.435090 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:29.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:29.531635 I | op-osd: updating OSD 5 on PVC "smithi007-lv-1-data-0jzl5l" 2022-03-10T02:00:29.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:29.531662 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0jzl5l" 2022-03-10T02:00:29.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:29.709103 I | op-mon: Monitors in quorum: [a b c] 2022-03-10T02:00:29.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:29.709146 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-10T02:00:29.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:29.747939 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 5 on PVC "smithi007-lv-1-data-0jzl5l" LastHeartbeatTime:2022-03-10 02:00:29.737426914 +0000 UTC m=+490.864926018 LastTransitionTime:2022-03-10 02:00:29.737426722 +0000 UTC m=+490.864925850}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:30.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:30.191711 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-10T02:00:30.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:30.236106 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:30.332 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:30.330386 I | op-osd: updating OSD 6 on PVC "smithi007-lv-3-data-0xslw4" 2022-03-10T02:00:30.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:30.330425 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0xslw4" 2022-03-10T02:00:30.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:30.546473 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 6 on PVC "smithi007-lv-3-data-0xslw4" LastHeartbeatTime:2022-03-10 02:00:30.538004415 +0000 UTC m=+491.665503528 LastTransitionTime:2022-03-10 02:00:30.538004118 +0000 UTC m=+491.665503283}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:30.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:30.700914 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-10T02:00:31.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.040075 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:31.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.122644 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-10T02:00:31.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.129252 I | op-osd: updating OSD 7 on PVC "smithi007-lv-4-data-0cfg7m" 2022-03-10T02:00:31.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.129273 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0cfg7m" 2022-03-10T02:00:31.358 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.355737 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 7 on PVC "smithi007-lv-4-data-0cfg7m" LastHeartbeatTime:2022-03-10 02:00:31.347646562 +0000 UTC m=+492.475145666 LastTransitionTime:2022-03-10 02:00:31.34764635 +0000 UTC m=+492.475145503}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:31.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.582142 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-10T02:00:31.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.582177 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-10T02:00:31.872 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.869721 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:31.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.931874 I | op-osd: updating OSD 8 on PVC "smithi184-lv-2-data-0l5x8k" 2022-03-10T02:00:31.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:31.931906 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-2-data-0l5x8k" 2022-03-10T02:00:32.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.057407 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-10T02:00:32.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.163513 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 8 on PVC "smithi184-lv-2-data-0l5x8k" LastHeartbeatTime:2022-03-10 02:00:32.154109872 +0000 UTC m=+493.281608958 LastTransitionTime:2022-03-10 02:00:32.154109748 +0000 UTC m=+493.281608859}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:32.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.551356 I | cephclient: successfully enabled msgr2 protocol 2022-03-10T02:00:32.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.551396 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-10T02:00:32.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.551405 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-10T02:00:32.638 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.636211 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:32.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.729226 I | op-osd: updating OSD 9 on PVC "smithi184-lv-1-data-096cgx" 2022-03-10T02:00:32.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.729254 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-1-data-096cgx" 2022-03-10T02:00:32.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.954065 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 9 on PVC "smithi184-lv-1-data-096cgx" LastHeartbeatTime:2022-03-10 02:00:32.944273913 +0000 UTC m=+494.071773054 LastTransitionTime:2022-03-10 02:00:32.94427367 +0000 UTC m=+494.071772834}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:32.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.966125 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-10T02:00:32.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.989859 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph Mgr(s) LastHeartbeatTime:2022-03-10 02:00:32.981581397 +0000 UTC m=+494.109080523 LastTransitionTime:2022-03-10 02:00:32.981581197 +0000 UTC m=+494.109080329}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:32.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.989883 I | op-mgr: start running mgr 2022-03-10T02:00:32.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:32.989901 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-10T02:00:33.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:33.341226 I | op-osd: OSD orchestration status for node smithi071-lv-1-data-0kfwx5 is "orchestrating" 2022-03-10T02:00:33.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:33.342880 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0kfwx5 is "orchestrating" 2022-03-10T02:00:33.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:33.563161 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-10T02:00:33.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:33.735611 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-10T02:00:33.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:33.929852 I | op-mgr: setting services to point to mgr "a" 2022-03-10T02:00:34.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.095289 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-10T02:00:34.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.095428 I | op-mgr: successful modules: balancer 2022-03-10T02:00:34.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.105329 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring Ceph OSDs LastHeartbeatTime:2022-03-10 02:00:34.095411347 +0000 UTC m=+495.222910454 LastTransitionTime:2022-03-10 02:00:34.095411115 +0000 UTC m=+495.222910254}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:34.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.105359 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-10T02:00:34.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.105368 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-10T02:00:34.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.156957 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-10T02:00:34.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172525 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-1" 2022-03-10T02:00:34.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172576 I | op-osd: OSD PVC "smithi007-lv-1-data-0jzl5l" already exists 2022-03-10T02:00:34.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172594 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-2" 2022-03-10T02:00:34.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172614 I | op-osd: OSD PVC "smithi007-lv-2-data-08blxh" already exists 2022-03-10T02:00:34.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172631 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-3" 2022-03-10T02:00:34.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172651 I | op-osd: OSD PVC "smithi007-lv-3-data-0xslw4" already exists 2022-03-10T02:00:34.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172667 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-4" 2022-03-10T02:00:34.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172684 I | op-osd: OSD PVC "smithi007-lv-4-data-0cfg7m" already exists 2022-03-10T02:00:34.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172696 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi071-lv-2" 2022-03-10T02:00:34.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172715 I | op-osd: OSD PVC "smithi071-lv-2-data-0lmhm6" already exists 2022-03-10T02:00:34.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172730 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi071-lv-3" 2022-03-10T02:00:34.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172749 I | op-osd: OSD PVC "smithi071-lv-3-data-0xjgfz" already exists 2022-03-10T02:00:34.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172763 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi071-lv-4" 2022-03-10T02:00:34.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172781 I | op-osd: OSD PVC "smithi071-lv-4-data-0q5d9z" already exists 2022-03-10T02:00:34.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172796 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi184-lv-1" 2022-03-10T02:00:34.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172816 I | op-osd: OSD PVC "smithi184-lv-1-data-096cgx" already exists 2022-03-10T02:00:34.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172832 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi184-lv-2" 2022-03-10T02:00:34.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172850 I | op-osd: OSD PVC "smithi184-lv-2-data-0l5x8k" already exists 2022-03-10T02:00:34.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172867 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi184-lv-3" 2022-03-10T02:00:34.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172890 I | op-osd: OSD PVC "smithi184-lv-3-data-09sg48" already exists 2022-03-10T02:00:34.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172906 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi184-lv-4" 2022-03-10T02:00:34.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172923 I | op-osd: OSD PVC "smithi184-lv-4-data-0n7czp" already exists 2022-03-10T02:00:34.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172936 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi071-lv-1" 2022-03-10T02:00:34.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.172954 I | op-osd: OSD PVC "smithi071-lv-1-data-0kfwx5" already exists 2022-03-10T02:00:34.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366424 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0jzl5l" 2022-03-10T02:00:34.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366443 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-1-data-0jzl5l" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366446 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-08blxh" 2022-03-10T02:00:34.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366449 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-2-data-08blxh" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366452 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0xslw4" 2022-03-10T02:00:34.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366454 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-3-data-0xslw4" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366457 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0cfg7m" 2022-03-10T02:00:34.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366461 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-4-data-0cfg7m" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366465 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-2-data-0lmhm6" 2022-03-10T02:00:34.377 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366469 I | op-osd: skipping OSD prepare job creation for PVC "smithi071-lv-2-data-0lmhm6" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366473 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-3-data-0xjgfz" 2022-03-10T02:00:34.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366476 I | op-osd: skipping OSD prepare job creation for PVC "smithi071-lv-3-data-0xjgfz" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366479 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-4-data-0q5d9z" 2022-03-10T02:00:34.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366482 I | op-osd: skipping OSD prepare job creation for PVC "smithi071-lv-4-data-0q5d9z" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366485 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-1-data-096cgx" 2022-03-10T02:00:34.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366488 I | op-osd: skipping OSD prepare job creation for PVC "smithi184-lv-1-data-096cgx" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366491 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-2-data-0l5x8k" 2022-03-10T02:00:34.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366494 I | op-osd: skipping OSD prepare job creation for PVC "smithi184-lv-2-data-0l5x8k" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366498 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-3-data-09sg48" 2022-03-10T02:00:34.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366502 I | op-osd: skipping OSD prepare job creation for PVC "smithi184-lv-3-data-09sg48" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366507 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-4-data-0n7czp" 2022-03-10T02:00:34.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366510 I | op-osd: skipping OSD prepare job creation for PVC "smithi184-lv-4-data-0n7czp" because OSD daemon using the PVC already exists 2022-03-10T02:00:34.381 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.366513 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:34.381 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.371398 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0kfwx5 is "starting" 2022-03-10T02:00:34.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.380458 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi071-lv-1-data-0kfwx5. Status={Conditions:[] StartTime:2022-03-10 02:00:24 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-10T02:00:34.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.380474 I | op-osd: started OSD provisioning job for PVC "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:34.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.380481 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-10T02:00:34.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.380484 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-10T02:00:34.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.382699 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0kfwx5 is "starting" 2022-03-10T02:00:34.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.484009 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:34.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.576599 I | op-osd: updating OSD 1 on PVC "smithi071-lv-2-data-0lmhm6" 2022-03-10T02:00:34.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.576637 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-2-data-0lmhm6" 2022-03-10T02:00:34.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:34.747014 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 1 on PVC "smithi071-lv-2-data-0lmhm6" LastHeartbeatTime:2022-03-10 02:00:34.736330777 +0000 UTC m=+495.863829864 LastTransitionTime:2022-03-10 02:00:34.736330597 +0000 UTC m=+495.863829712}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:35.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:35.105413 I | op-mgr: successful modules: dashboard 2022-03-10T02:00:35.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:35.132571 I | op-mgr: successful modules: prometheus 2022-03-10T02:00:35.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:35.133285 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-10T02:00:35.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:35.234918 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:35.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:35.367293 I | op-osd: updating OSD 10 on PVC "smithi184-lv-3-data-09sg48" 2022-03-10T02:00:35.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:35.367330 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-3-data-09sg48" 2022-03-10T02:00:35.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:35.546338 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 10 on PVC "smithi184-lv-3-data-09sg48" LastHeartbeatTime:2022-03-10 02:00:35.537080708 +0000 UTC m=+496.664579814 LastTransitionTime:2022-03-10 02:00:35.537080452 +0000 UTC m=+496.664579631}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:36.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:36.044647 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:36.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:36.131473 I | op-osd: updating OSD 11 on PVC "smithi184-lv-4-data-0n7czp" 2022-03-10T02:00:36.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:36.131509 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-4-data-0n7czp" 2022-03-10T02:00:36.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:36.347799 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 11 on PVC "smithi184-lv-4-data-0n7czp" LastHeartbeatTime:2022-03-10 02:00:36.338034182 +0000 UTC m=+497.465533323 LastTransitionTime:2022-03-10 02:00:36.338033959 +0000 UTC m=+497.465533120}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:36.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:36.765162 I | op-mgr: successful modules: orchestrator modules 2022-03-10T02:00:36.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:36.837474 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:36.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:36.930769 I | op-osd: updating OSD 2 on PVC "smithi071-lv-3-data-0xjgfz" 2022-03-10T02:00:36.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:36.930799 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-3-data-0xjgfz" 2022-03-10T02:00:37.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:37.161442 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 2 on PVC "smithi071-lv-3-data-0xjgfz" LastHeartbeatTime:2022-03-10 02:00:37.154195481 +0000 UTC m=+498.281694636 LastTransitionTime:2022-03-10 02:00:37.154195214 +0000 UTC m=+498.281694403}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:37.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:37.637788 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:37.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:37.731884 I | op-osd: updating OSD 3 on PVC "smithi071-lv-4-data-0q5d9z" 2022-03-10T02:00:37.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:37.731924 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-4-data-0q5d9z" 2022-03-10T02:00:37.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:37.949729 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 3 on PVC "smithi071-lv-4-data-0q5d9z" LastHeartbeatTime:2022-03-10 02:00:37.937405194 +0000 UTC m=+499.064904330 LastTransitionTime:2022-03-10 02:00:37.937404956 +0000 UTC m=+499.064904124}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:38.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:38.437247 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:38.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:38.531310 I | op-osd: updating OSD 4 on PVC "smithi007-lv-2-data-08blxh" 2022-03-10T02:00:38.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:38.531339 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-08blxh" 2022-03-10T02:00:38.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:38.746730 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 4 on PVC "smithi007-lv-2-data-08blxh" LastHeartbeatTime:2022-03-10 02:00:38.738591205 +0000 UTC m=+499.866090310 LastTransitionTime:2022-03-10 02:00:38.738591005 +0000 UTC m=+499.866090153}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:39.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:39.167967 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0kfwx5 is "completed" 2022-03-10T02:00:39.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:39.168000 I | op-osd: creating OSD 0 on PVC "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:39.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:39.168017 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:39.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:39.168104 I | op-osd: OSD orchestration status for PVC smithi071-lv-1-data-0kfwx5 is "completed" 2022-03-10T02:00:39.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:39.168146 I | op-osd: creating OSD 0 on PVC "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:39.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:39.168169 I | op-osd: OSD will have its main bluestore block on "smithi071-lv-1-data-0kfwx5" 2022-03-10T02:00:39.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:39.373152 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi071-lv-1-data-0kfwx5" LastHeartbeatTime:2022-03-10 02:00:39.349713127 +0000 UTC m=+500.477212249 LastTransitionTime:2022-03-10 02:00:39.349712858 +0000 UTC m=+500.477212034}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:39.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:39.561748 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi071-lv-1-data-0kfwx5" LastHeartbeatTime:2022-03-10 02:00:39.551948653 +0000 UTC m=+500.679447754 LastTransitionTime:2022-03-10 02:00:39.551948455 +0000 UTC m=+500.679447582}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:40.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.013960 E | op-osd: failed to create OSD 0 on PVC "smithi071-lv-1-data-0kfwx5": failed to create deployment for OSD 0 on PVC "smithi071-lv-1-data-0kfwx5": deployments.apps "rook-ceph-osd-0" already exists 2022-03-10T02:00:40.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.116933 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:40.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.201453 I | op-osd: updating OSD 5 on PVC "smithi007-lv-1-data-0jzl5l" 2022-03-10T02:00:40.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.201480 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-0jzl5l" 2022-03-10T02:00:40.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.303579 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-10T02:00:40.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.303597 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-10T02:00:40.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.314001 E | ceph-spec: failed to update cluster condition to {Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully LastHeartbeatTime:2022-03-10 02:00:40.303658704 +0000 UTC m=+501.431157784 LastTransitionTime:2022-03-10 01:57:07 +0000 UTC}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:40.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.391653 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 5 on PVC "smithi007-lv-1-data-0jzl5l" LastHeartbeatTime:2022-03-10 02:00:40.381871718 +0000 UTC m=+501.509370884 LastTransitionTime:2022-03-10 02:00:40.381871394 +0000 UTC m=+501.509370600}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:40.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.837158 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:40.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.932586 I | op-osd: updating OSD 6 on PVC "smithi007-lv-3-data-0xslw4" 2022-03-10T02:00:40.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:40.932619 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-0xslw4" 2022-03-10T02:00:41.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:41.147242 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 6 on PVC "smithi007-lv-3-data-0xslw4" LastHeartbeatTime:2022-03-10 02:00:41.137920221 +0000 UTC m=+502.265419333 LastTransitionTime:2022-03-10 02:00:41.137920053 +0000 UTC m=+502.265419185}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:41.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:41.638459 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:41.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:41.747722 I | op-osd: updating OSD 7 on PVC "smithi007-lv-4-data-0cfg7m" 2022-03-10T02:00:41.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:41.747754 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0cfg7m" 2022-03-10T02:00:41.964 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:41.963412 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 7 on PVC "smithi007-lv-4-data-0cfg7m" LastHeartbeatTime:2022-03-10 02:00:41.953949226 +0000 UTC m=+503.081448359 LastTransitionTime:2022-03-10 02:00:41.953948963 +0000 UTC m=+503.081448135}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:42.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:42.437546 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:42.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:42.533060 I | op-osd: updating OSD 8 on PVC "smithi184-lv-2-data-0l5x8k" 2022-03-10T02:00:42.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:42.533099 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-2-data-0l5x8k" 2022-03-10T02:00:42.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:42.745852 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 8 on PVC "smithi184-lv-2-data-0l5x8k" LastHeartbeatTime:2022-03-10 02:00:42.738273012 +0000 UTC m=+503.865772140 LastTransitionTime:2022-03-10 02:00:42.738272818 +0000 UTC m=+503.865771954}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:43.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:43.237664 I | op-osd: skipping osd checks for ok-to-stop 2022-03-10T02:00:43.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:43.355927 I | op-osd: updating OSD 9 on PVC "smithi184-lv-1-data-096cgx" 2022-03-10T02:00:43.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:43.355965 I | op-osd: OSD will have its main bluestore block on "smithi184-lv-1-data-096cgx" 2022-03-10T02:00:43.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:43.549772 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 9 on PVC "smithi184-lv-1-data-096cgx" LastHeartbeatTime:2022-03-10 02:00:43.539914555 +0000 UTC m=+504.667413691 LastTransitionTime:2022-03-10 02:00:43.539914325 +0000 UTC m=+504.667413471}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:43.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:43.947750 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:False Reason:ClusterProgressing Message:failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-10T02:00:43.949 INFO:tasks.rook.operator.smithi007.stdout:failed to create OSD 0 on PVC "smithi071-lv-1-data-0kfwx5": failed to create deployment for OSD 0 on PVC "smithi071-lv-1-data-0kfwx5": deployments.apps "rook-ceph-osd-0" already exists LastHeartbeatTime:2022-03-10 02:00:43.937079569 +0000 UTC m=+505.064578685 LastTransitionTime:2022-03-10 02:00:43.937079332 +0000 UTC m=+505.064578513}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:00:43.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:00:43.947794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-10T02:00:43.950 INFO:tasks.rook.operator.smithi007.stdout:failed to create OSD 0 on PVC "smithi071-lv-1-data-0kfwx5": failed to create deployment for OSD 0 on PVC "smithi071-lv-1-data-0kfwx5": deployments.apps "rook-ceph-osd-0" already exists 2022-03-10T02:00:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T02:00:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T02:00:44.740 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-10T02:00:44.741 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-10T02:00:44.742 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T02:00:44.742 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-10T02:00:44.816 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T02:00:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-10T02:00:45.285 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-10T02:01:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-10T02:01:15.287 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-10T02:01:15.288 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-10T02:01:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-10T02:01:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-10T02:01:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-10T02:01:15.365 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-10T02:01:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=12 2022-03-10T02:01:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-10T02:01:45.818 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 12 '!=' 12 ']' 2022-03-10T02:01:45.820 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-10T02:01:45.833 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch status 2022-03-10T02:01:46.598 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: rook 2022-03-10T02:01:46.598 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2022-03-10T02:01:46.619 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ps 2022-03-10T02:01:47.190 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-10T02:01:47.190 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi007 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-10T02:01:47.190 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi071 running (4m) 0s ago 4m - - 4b871cde5e1a 2022-03-10T02:01:47.191 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi184 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.191 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.a smithi184 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-10T02:01:47.191 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi071 running (7m) 0s ago 7m - - 4b871cde5e1a 2022-03-10T02:01:47.191 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi184 running (6m) 0s ago 6m - - 4b871cde5e1a 2022-03-10T02:01:47.192 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (5m) 0s ago 5m - - 4b871cde5e1a 2022-03-10T02:01:47.192 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi071 running (68s) 0s ago 68s - - 4b871cde5e1a 2022-03-10T02:01:47.192 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi071 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.192 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 smithi184 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.193 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 smithi184 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.193 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi071 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.193 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi071 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.193 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi007 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.194 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi007 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.194 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi007 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.194 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi007 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.194 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 smithi184 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.195 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 smithi184 running (3m) 0s ago 3m - - 4b871cde5e1a 2022-03-10T02:01:47.209 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls 2022-03-10T02:01:47.927 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-10T02:01:47.927 INFO:teuthology.orchestra.run.smithi007.stdout:crash 3/3 0s ago 4m * 2022-03-10T02:01:47.927 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 1/1 0s ago 5m count:1 2022-03-10T02:01:47.928 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 0s ago 7m count:3 2022-03-10T02:01:47.928 INFO:teuthology.orchestra.run.smithi007.stdout:osd 24 0s ago 3m 2022-03-10T02:01:47.928 INFO:teuthology.orchestra.run.smithi007.stdout:osd.all-available-devices 0 0s ago - * 2022-03-10T02:01:47.951 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch host ls 2022-03-10T02:01:48.495 INFO:teuthology.orchestra.run.smithi007.stdout:HOST ADDR LABELS STATUS 2022-03-10T02:01:48.495 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 172.21.15.7/smithi007 2022-03-10T02:01:48.496 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 172.21.15.71/smithi071 2022-03-10T02:01:48.496 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 172.21.15.184/smithi184 2022-03-10T02:01:48.496 INFO:teuthology.orchestra.run.smithi007.stdout:3 hosts in cluster 2022-03-10T02:01:48.513 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch device ls 2022-03-10T02:01:49.071 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-10T02:01:49.072 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T02:01:49.072 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T02:01:49.072 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T02:01:49.072 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T02:01:49.073 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T02:01:49.073 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T02:01:49.073 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T02:01:49.074 INFO:teuthology.orchestra.run.smithi007.stdout:smithi071 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T02:01:49.074 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-10T02:01:49.074 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-10T02:01:49.074 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-10T02:01:49.075 INFO:teuthology.orchestra.run.smithi007.stdout:smithi184 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-10T02:01:49.092 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch apply rgw foo 2022-03-10T02:01:49.664 INFO:teuthology.orchestra.run.smithi007.stdout:Created 2022-03-10T02:01:49.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:49.666319 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-10T02:01:49.687 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch apply mds foo 2022-03-10T02:01:49.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:49.709736 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:01:49.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:49.709804 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T02:01:50.311 INFO:teuthology.orchestra.run.smithi007.stdout:Created 2022-03-10T02:01:50.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:50.312202 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T02:01:50.336 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch apply rbd-mirror 2022-03-10T02:01:50.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:50.345736 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:01:50.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:50.351155 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:01:50.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:50.351246 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T02:01:51.137 INFO:teuthology.orchestra.run.smithi007.stdout:Success 2022-03-10T02:01:51.158 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch apply nfs foo --port 12777 2022-03-10T02:01:51.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:51.217881 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:01:51.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:51.217936 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3... 2022-03-10T02:01:55.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:55.584424 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T02:01:55.761 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-10T02:01:55.761 INFO:teuthology.orchestra.run.smithi007.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-10T02:01:55.783 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-10T02:01:55.795 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:01:56.629 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:01:56.629 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:50.000000Z", "last_refresh": "2022-03-10T02:01:56.383617Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:01:56.383617Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:01:56.383617Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:01:56.383617Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:01:56.383617Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:01:56.383617Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-10T02:01:56.383617Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:01:56.383617Z", "running": 0, "size": 1}}] 2022-03-10T02:01:56.654 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch rm mds.foo 2022-03-10T02:01:56.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:56.907366 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T02:01:57.155 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:57.155842 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-10T02:01:57.165 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephfilesystems/foo 2022-03-10T02:01:57.191 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch rm rbd-mirror 2022-03-10T02:01:57.688 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-10T02:01:57.710 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch rm rgw.foo 2022-03-10T02:01:57.713 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:57.714582 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-10T02:01:58.263 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephobjectstores/foo 2022-03-10T02:01:58.285 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:01:58.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:58.555668 I | ceph-object-controller: reconciling object store deployments 2022-03-10T02:01:58.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:58.595177 I | ceph-object-controller: ceph object store gateway service running at 10.101.218.200 2022-03-10T02:01:58.596 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:58.595199 I | ceph-object-controller: reconciling object store pools 2022-03-10T02:01:58.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:58.661955 I | op-mds: setting mds config flags 2022-03-10T02:01:58.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:58.662003 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T02:01:58.968 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:01:58.969 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:50.000000Z", "last_refresh": "2022-03-10T02:01:58.817897Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:01:58.817897Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:01:58.817897Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:01:58.817897Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:01:58.817897Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:01:58.817897Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:01:58.817897Z", "running": 0, "size": 1}}] 2022-03-10T02:01:59.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:01:59.424456 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-10T02:02:00.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:00.046273 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T02:02:00.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:00.057981 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-10T02:02:00.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:00.247155 I | ceph-spec: detected ceph image version: "17.0.0-11025 quincy" 2022-03-10T02:02:00.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:00.652938 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-10T02:02:00.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:00.652968 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-10T02:02:00.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:00.652982 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-10T02:02:00.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:00.990730 I | op-mds: setting mds config flags 2022-03-10T02:02:00.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:00.990763 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T02:02:02.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:02.109017 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-10T02:02:02.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:02.289576 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-10T02:02:02.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:02.321443 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-10T02:02:02.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:02.623619 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi184": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:02:02.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:02.768055 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi184": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:02:04.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:04.952680 I | ceph-file-controller: creating filesystem "foo" 2022-03-10T02:02:08.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:08.152378 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-10T02:02:08.160 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:08.161260 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-10T02:02:09.009 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:02:09.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:09.207547 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-10T02:02:09.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:09.209381 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-10T02:02:09.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:09.211781 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-10T02:02:09.715 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:02:09.716 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:50.000000Z", "last_refresh": "2022-03-10T02:02:09.537225Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:02:09.537225Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:02:09.537225Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:02:09.537225Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:02:09.537225Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:02:09.537225Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:02:09.537225Z", "running": 0, "size": 1}}] 2022-03-10T02:02:10.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:10.152921 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-10T02:02:10.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:10.168088 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:11.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:11.178754 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-10T02:02:14.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:14.168289 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:14.159+0000 7fad27c08380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:14.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:14.159+0000 7fad27c08380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:14.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:14.159+0000 7fad27c08380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:14.169 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:14.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:14.184665 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:14.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:14.816335 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:14.807+0000 7f12f0400380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:14.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:14.807+0000 7f12f0400380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:14.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:14.807+0000 7f12f0400380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:14.986 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:14.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:14.828071 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:15.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:15.530060 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-10T02:02:16.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:16.082889 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:16.075+0000 7f8a6975f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:16.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:16.075+0000 7f8a6975f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:16.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:16.075+0000 7f8a6975f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:16.084 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:16.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:16.118684 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:16.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:16.830354 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-10T02:02:16.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:16.890788 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:16.883+0000 7f2153252380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:16.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:16.883+0000 7f2153252380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:16.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:16.883+0000 7f2153252380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:16.891 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:16.919 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:16.919909 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:17.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:17.523984 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:17.515+0000 7f2d4db92380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:17.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:17.515+0000 7f2d4db92380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:17.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:17.515+0000 7f2d4db92380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:17.524 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:17.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:17.551791 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:18.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:18.152519 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:18.143+0000 7fd68debe380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:18.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:18.143+0000 7fd68debe380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:18.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:18.143+0000 7fd68debe380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:18.153 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:18.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:18.161780 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:18.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:18.750668 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:18.743+0000 7f19c6cf9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:18.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:18.743+0000 7f19c6cf9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:18.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:18.743+0000 7f19c6cf9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:18.751 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:18.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:18.767736 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:19.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:19.012779 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-10T02:02:19.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:19.012799 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-10T02:02:19.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:19.501956 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:19.490+0000 7f133e1bb380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:19.504 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:19.490+0000 7f133e1bb380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:19.504 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:19.490+0000 7f133e1bb380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:19.504 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:19.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:19.522826 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:19.746 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:02:20.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:20.116645 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:02:20.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:20.120903 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:20.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:20.123016 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-10T02:02:20.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:20.212064 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:20.202+0000 7f1447ea9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:20.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:20.202+0000 7f1447ea9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:20.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:20.202+0000 7f1447ea9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:20.213 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:20.488 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:02:20.489 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:02:20.311365Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:02:20.311365Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:02:20.311365Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:02:20.311365Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:02:20.311365Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:02:20.311365Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:02:20.311365Z", "running": 0, "size": 1}}] 2022-03-10T02:02:20.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:20.787955 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:20.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:20.980794 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-10T02:02:21.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:21.471179 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-10T02:02:21.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:21.471209 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-10T02:02:21.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:21.471216 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-10T02:02:21.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:21.472947 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:21.462+0000 7fbe94bf3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:21.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:21.462+0000 7fbe94bf3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:21.475 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:21.462+0000 7fbe94bf3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:21.475 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:22.056 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:22.056463 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-10T02:02:22.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:22.454807 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-10T02:02:23.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:23.062901 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-10T02:02:23.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:23.062944 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-10T02:02:23.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:23.062953 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-10T02:02:23.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:23.650947 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-10T02:02:23.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:23.650986 I | ceph-file-controller: downing filesystem "foo" 2022-03-10T02:02:24.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:24.844858 I | ceph-file-controller: downed filesystem "foo" 2022-03-10T02:02:26.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:26.605316 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:27.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:27.517827 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:27.506+0000 7f1dc4405380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:27.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:27.506+0000 7f1dc4405380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:27.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:27.506+0000 7f1dc4405380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:27.518 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:28.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:28.684408 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-10T02:02:28.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:28.684433 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-10T02:02:30.511 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:02:31.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:31.583009 I | cephclient: purge completed for pool "foo-metadata" 2022-03-10T02:02:31.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:31.974032 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-10T02:02:31.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:31.974070 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-10T02:02:34.067 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:02:34.068 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:02:33.844196Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:02:33.844196Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:02:33.844196Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:02:33.844196Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:02:33.844196Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:02:33.844196Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:02:33.844196Z", "running": 0, "size": 1}}] 2022-03-10T02:02:34.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:34.840598 I | cephclient: purge completed for pool "foo-data0" 2022-03-10T02:02:34.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:34.840691 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-10T02:02:34.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:34.893501 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-10T02:02:34.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:34.900690 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-10T02:02:34.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:34.900790 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-10T02:02:34.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:34.900912 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-10T02:02:37.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:37.682339 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi071": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:02:37.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:37.839830 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:38.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:38.487960 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:38.474+0000 7fc1792aa380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:38.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:38.474+0000 7fc1792aa380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:38.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:38.474+0000 7fc1792aa380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:38.489 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:02:44.091 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:02:45.605 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:02:45.606 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:02:45.268603Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:02:45.268603Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:02:45.268603Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:02:45.268603Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:02:45.268603Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:02:45.268603Z", "running": 0, "size": 1}}] 2022-03-10T02:02:55.629 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:02:56.328 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:02:56.328 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:02:56.162821Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:02:56.162821Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:02:56.162821Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:02:56.162821Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:02:56.162821Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:02:56.162821Z", "running": 0, "size": 1}}] 2022-03-10T02:02:59.026 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:59.026487 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:02:59.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:02:59.662893 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:02:59.650+0000 7f98a427f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:02:59.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:59.650+0000 7f98a427f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:02:59.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:02:59.650+0000 7f98a427f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:02:59.665 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:03:06.351 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:03:07.106 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:03:07.107 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:03:06.911172Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:03:06.911172Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:03:06.911172Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:03:06.911172Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:03:06.911172Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:03:06.911172Z", "running": 0, "size": 1}}] 2022-03-10T02:03:17.132 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:03:17.825 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:03:17.826 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:03:17.638641Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:03:17.638641Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:03:17.638641Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:03:17.638641Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:03:17.638641Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:03:17.638641Z", "running": 0, "size": 1}}] 2022-03-10T02:03:27.853 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:03:28.579 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:03:28.580 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:03:28.420042Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:03:28.420042Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:03:28.420042Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:03:28.420042Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:03:28.420042Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:03:28.420042Z", "running": 0, "size": 1}}] 2022-03-10T02:03:38.602 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:03:39.318 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:03:39.319 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:03:39.136863Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:03:39.136863Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:03:39.136863Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:03:39.136863Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:03:39.136863Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:03:39.136863Z", "running": 0, "size": 1}}] 2022-03-10T02:03:40.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:03:40.628971 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:03:41.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:03:41.373428 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:03:41.365+0000 7f5f858d5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:03:41.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:03:41.365+0000 7f5f858d5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:03:41.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:03:41.365+0000 7f5f858d5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:03:41.375 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:03:49.343 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:03:50.099 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:03:50.099 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:03:49.856748Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:03:49.856748Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:03:49.856748Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:03:49.856748Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:03:49.856748Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:03:49.856748Z", "running": 0, "size": 1}}] 2022-03-10T02:04:00.125 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:04:00.869 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:04:00.869 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:04:00.686750Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:04:00.686750Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:04:00.686750Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:04:00.686750Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:04:00.686750Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:04:00.686750Z", "running": 0, "size": 1}}] 2022-03-10T02:04:06.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:04:06.737816 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi184": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:04:06.972 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:04:06.971095 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi184": the object has been modified; please apply your changes to the latest version and try again 2022-03-10T02:04:10.894 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:04:11.667 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:04:11.668 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:04:11.450012Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:04:11.450012Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:04:11.450012Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:04:11.450012Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:04:11.450012Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:04:11.450012Z", "running": 0, "size": 1}}] 2022-03-10T02:04:21.690 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:04:22.413 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:04:22.413 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:04:22.243592Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:04:22.243592Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:04:22.243592Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:04:22.243592Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:04:22.243592Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:04:22.243592Z", "running": 0, "size": 1}}] 2022-03-10T02:04:32.436 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:04:33.131 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:04:33.132 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:04:32.960209Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:04:32.960209Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:04:32.960209Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:04:32.960209Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:04:32.960209Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:04:32.960209Z", "running": 0, "size": 1}}] 2022-03-10T02:04:43.152 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:04:43.874 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:04:43.875 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:04:43.677553Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:04:43.677553Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:04:43.677553Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:04:43.677553Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:04:43.677553Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:04:43.677553Z", "running": 0, "size": 1}}] 2022-03-10T02:04:53.900 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:04:54.604 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:04:54.605 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:04:54.423430Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:04:54.423430Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:04:54.423430Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:04:54.423430Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:04:54.423430Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:04:54.423430Z", "running": 0, "size": 1}}] 2022-03-10T02:05:03.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:05:03.303166 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:05:03.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:05:03.991098 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:05:03.980+0000 7fe46d2f9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:05:03.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:05:03.980+0000 7fe46d2f9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:05:03.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:05:03.980+0000 7fe46d2f9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:05:03.996 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:05:04.626 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:05:05.602 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:05:05.603 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:05:05.167201Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:05:05.167201Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:05:05.167201Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:05:05.167201Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:05:05.167201Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:05:05.167201Z", "running": 0, "size": 1}}] 2022-03-10T02:05:15.623 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:05:16.382 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:05:16.383 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:05:16.152896Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:05:16.152896Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:05:16.152896Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:05:16.152896Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:05:16.152896Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:05:16.152896Z", "running": 0, "size": 1}}] 2022-03-10T02:05:26.404 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:05:27.086 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:05:27.086 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:05:26.914877Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:05:26.914877Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:05:26.914877Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:05:26.914877Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:05:26.914877Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:05:26.914877Z", "running": 0, "size": 1}}] 2022-03-10T02:05:37.109 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:05:37.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:05:37.776 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:05:37.602523Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:05:37.602523Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:05:37.602523Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:05:37.602523Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:05:37.602523Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:05:37.602523Z", "running": 0, "size": 1}}] 2022-03-10T02:05:47.799 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:05:48.633 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:05:48.633 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:05:48.317182Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:05:48.317182Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:05:48.317182Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:05:48.317182Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:05:48.317182Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:05:48.317182Z", "running": 0, "size": 1}}] 2022-03-10T02:05:58.656 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:05:59.399 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:05:59.400 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:05:59.230263Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:05:59.230263Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:05:59.230263Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:05:59.230263Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:05:59.230263Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:05:59.230263Z", "running": 0, "size": 1}}] 2022-03-10T02:06:09.422 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:06:10.087 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:06:10.087 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:06:09.911485Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:06:09.911485Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:06:09.911485Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:06:09.911485Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:06:09.911485Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:06:09.911485Z", "running": 0, "size": 1}}] 2022-03-10T02:06:20.111 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:06:20.782 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:06:20.782 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:06:20.623248Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:06:20.623248Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:06:20.623248Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:06:20.623248Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:06:20.623248Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:06:20.623248Z", "running": 0, "size": 1}}] 2022-03-10T02:06:30.803 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:06:31.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:06:31.529 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:06:31.338827Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:06:31.338827Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:06:31.338827Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:06:31.338827Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:06:31.338827Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:06:31.338827Z", "running": 0, "size": 1}}] 2022-03-10T02:06:41.558 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:06:42.352 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:06:42.353 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:06:42.163179Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:06:42.163179Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:06:42.163179Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:06:42.163179Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:06:42.163179Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:06:42.163179Z", "running": 0, "size": 1}}] 2022-03-10T02:06:52.376 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:06:53.075 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:06:53.075 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:06:52.900935Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:06:52.900935Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:06:52.900935Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:06:52.900935Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:06:52.900935Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:06:52.900935Z", "running": 0, "size": 1}}] 2022-03-10T02:07:03.097 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:07:03.820 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:07:03.821 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:07:03.646626Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:07:03.646626Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:07:03.646626Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:07:03.646626Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:07:03.646626Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:07:03.646626Z", "running": 0, "size": 1}}] 2022-03-10T02:07:13.844 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:07:14.550 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:07:14.551 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:07:14.365620Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:07:14.365620Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:07:14.365620Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:07:14.365620Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:07:14.365620Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:07:14.365620Z", "running": 0, "size": 1}}] 2022-03-10T02:07:24.573 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:07:25.258 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:07:25.259 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:07:25.064963Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:07:25.064963Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:07:25.064963Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:07:25.064963Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:07:25.064963Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:07:25.064963Z", "running": 0, "size": 1}}] 2022-03-10T02:07:35.282 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:07:35.978 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:07:35.979 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:07:35.795049Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:07:35.795049Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:07:35.795049Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:07:35.795049Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:07:35.795049Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:07:35.795049Z", "running": 0, "size": 1}}] 2022-03-10T02:07:46.001 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:07:46.756 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:07:46.756 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:07:46.536329Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:07:46.536329Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:07:46.536329Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:07:46.536329Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:07:46.536329Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:07:46.536329Z", "running": 0, "size": 1}}] 2022-03-10T02:07:47.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:07:47.843397 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:07:48.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:07:48.569358 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:07:48.555+0000 7f25aaa45380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:07:48.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:07:48.555+0000 7f25aaa45380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:07:48.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:07:48.555+0000 7f25aaa45380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:07:48.587 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:07:56.782 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:07:57.479 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:07:57.479 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:07:57.295717Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:07:57.295717Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:07:57.295717Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:07:57.295717Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:07:57.295717Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:07:57.295717Z", "running": 0, "size": 1}}] 2022-03-10T02:08:07.504 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:08:08.163 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:08:08.164 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:08:07.989222Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:08:07.989222Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:08:07.989222Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:08:07.989222Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:08:07.989222Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:08:07.989222Z", "running": 0, "size": 1}}] 2022-03-10T02:08:18.184 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:08:18.937 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:08:18.937 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:08:18.693207Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:08:18.693207Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:08:18.693207Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:08:18.693207Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:08:18.693207Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:08:18.693207Z", "running": 0, "size": 1}}] 2022-03-10T02:08:28.960 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:08:29.660 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:08:29.660 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:08:29.476984Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:08:29.476984Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:08:29.476984Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:08:29.476984Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:08:29.476984Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:08:29.476984Z", "running": 0, "size": 1}}] 2022-03-10T02:08:39.685 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:08:40.439 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:08:40.439 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:08:40.254520Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:08:40.254520Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:08:40.254520Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:08:40.254520Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:08:40.254520Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:08:40.254520Z", "running": 0, "size": 1}}] 2022-03-10T02:08:50.464 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:08:51.268 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:08:51.268 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:08:51.067151Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:08:51.067151Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:08:51.067151Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:08:51.067151Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:08:51.067151Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:08:51.067151Z", "running": 0, "size": 1}}] 2022-03-10T02:09:01.296 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:09:02.048 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:09:02.049 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:09:01.824426Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:09:01.824426Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:09:01.824426Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:09:01.824426Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:09:01.824426Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:09:01.824426Z", "running": 0, "size": 1}}] 2022-03-10T02:09:12.071 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:09:12.762 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:09:12.763 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:09:12.578122Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:09:12.578122Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:09:12.578122Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:09:12.578122Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:09:12.578122Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:09:12.578122Z", "running": 0, "size": 1}}] 2022-03-10T02:09:22.779 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:09:23.499 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:09:23.500 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:09:23.319131Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:09:23.319131Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:09:23.319131Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:09:23.319131Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:09:23.319131Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:09:23.319131Z", "running": 0, "size": 1}}] 2022-03-10T02:09:33.520 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:09:34.328 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:09:34.328 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:09:34.041782Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:09:34.041782Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:09:34.041782Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:09:34.041782Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:09:34.041782Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:09:34.041782Z", "running": 0, "size": 1}}] 2022-03-10T02:09:44.350 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:09:45.031 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:09:45.031 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:09:44.853603Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:09:44.853603Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:09:44.853603Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:09:44.853603Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:09:44.853603Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:09:44.853603Z", "running": 0, "size": 1}}] 2022-03-10T02:09:55.072 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:09:55.765 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:09:55.765 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:09:55.579842Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:09:55.579842Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:09:55.579842Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:09:55.579842Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:09:55.579842Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:09:55.579842Z", "running": 0, "size": 1}}] 2022-03-10T02:10:05.784 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:10:06.819 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:10:06.819 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:10:06.308500Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:10:06.308500Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:10:06.308500Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:10:06.308500Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:10:06.308500Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:10:06.308500Z", "running": 0, "size": 1}}] 2022-03-10T02:10:16.866 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:10:17.515 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:10:17.516 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:10:17.338100Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:10:17.338100Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:10:17.338100Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:10:17.338100Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:10:17.338100Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:10:17.338100Z", "running": 0, "size": 1}}] 2022-03-10T02:10:27.540 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:10:28.237 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:10:28.237 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:10:28.051965Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:10:28.051965Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:10:28.051965Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:10:28.051965Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:10:28.051965Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:10:28.051965Z", "running": 0, "size": 1}}] 2022-03-10T02:10:38.261 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:10:39.133 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:10:39.133 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:10:38.948556Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:10:38.948556Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:10:38.948556Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:10:38.948556Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:10:38.948556Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:10:38.948556Z", "running": 0, "size": 1}}] 2022-03-10T02:10:49.163 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:10:49.872 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:10:49.872 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:10:49.696794Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:10:49.696794Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:10:49.696794Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:10:49.696794Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:10:49.696794Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:10:49.696794Z", "running": 0, "size": 1}}] 2022-03-10T02:10:59.889 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:11:00.613 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:11:00.614 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:11:00.408558Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:11:00.408558Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:11:00.408558Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:11:00.408558Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:11:00.408558Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:11:00.408558Z", "running": 0, "size": 1}}] 2022-03-10T02:11:10.644 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:11:11.332 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:11:11.333 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:11:11.143748Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:11:11.143748Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:11:11.143748Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:11:11.143748Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:11:11.143748Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:11:11.143748Z", "running": 0, "size": 1}}] 2022-03-10T02:11:21.357 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:11:22.051 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:11:22.051 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:11:21.865446Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:11:21.865446Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:11:21.865446Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:11:21.865446Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:11:21.865446Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:11:21.865446Z", "running": 0, "size": 1}}] 2022-03-10T02:11:32.076 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:11:32.754 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:11:32.755 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:11:32.568172Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:11:32.568172Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:11:32.568172Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:11:32.568172Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:11:32.568172Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:11:32.568172Z", "running": 0, "size": 1}}] 2022-03-10T02:11:42.778 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:11:43.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:11:43.462 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:11:43.282750Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:11:43.282750Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:11:43.282750Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:11:43.282750Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:11:43.282750Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:11:43.282750Z", "running": 0, "size": 1}}] 2022-03-10T02:11:53.484 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:11:54.178 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:11:54.178 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:11:53.991453Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:11:53.991453Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:11:53.991453Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:11:53.991453Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:11:53.991453Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:11:53.991453Z", "running": 0, "size": 1}}] 2022-03-10T02:12:04.202 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:12:04.927 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:12:04.927 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:12:04.737642Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:12:04.737642Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:12:04.737642Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:12:04.737642Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:12:04.737642Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:12:04.737642Z", "running": 0, "size": 1}}] 2022-03-10T02:12:14.946 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:12:15.692 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:12:15.692 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:12:15.440616Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:12:15.440616Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:12:15.440616Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:12:15.440616Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:12:15.440616Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:12:15.440616Z", "running": 0, "size": 1}}] 2022-03-10T02:12:25.717 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:12:26.564 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:12:26.565 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:12:26.358104Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:12:26.358104Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:12:26.358104Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:12:26.358104Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:12:26.358104Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:12:26.358104Z", "running": 0, "size": 1}}] 2022-03-10T02:12:36.590 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:12:39.776 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:12:39.776 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:12:39.503339Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:12:39.503339Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:12:39.503339Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:12:39.503339Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:12:39.503339Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:12:39.503339Z", "running": 0, "size": 1}}] 2022-03-10T02:12:49.806 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:12:50.513 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:12:50.513 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:12:50.341220Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:12:50.341220Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:12:50.341220Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:12:50.341220Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:12:50.341220Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:12:50.341220Z", "running": 0, "size": 1}}] 2022-03-10T02:13:00.533 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:13:01.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:13:01.223 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:13:01.022372Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:13:01.022372Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:13:01.022372Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:13:01.022372Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:13:01.022372Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:13:01.022372Z", "running": 0, "size": 1}}] 2022-03-10T02:13:11.257 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:13:11.962 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:13:11.962 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:13:11.780997Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:13:11.780997Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:13:11.780997Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:13:11.780997Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:13:11.780997Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:13:11.780997Z", "running": 0, "size": 1}}] 2022-03-10T02:13:16.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:13:16.265280 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:13:16.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:13:16.939845 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:13:16.931+0000 7f7a0d6c7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:13:16.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:13:16.931+0000 7f7a0d6c7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:13:16.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:13:16.931+0000 7f7a0d6c7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:13:16.957 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:13:21.987 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:13:22.702 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:13:22.703 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:13:22.483718Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:13:22.483718Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:13:22.483718Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:13:22.483718Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:13:22.483718Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:13:22.483718Z", "running": 0, "size": 1}}] 2022-03-10T02:13:32.727 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:13:33.438 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:13:33.438 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:13:33.254327Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:13:33.254327Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:13:33.254327Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:13:33.254327Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:13:33.254327Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:13:33.254327Z", "running": 0, "size": 1}}] 2022-03-10T02:13:43.458 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:13:44.149 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:13:44.150 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:13:43.953465Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:13:43.953465Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:13:43.953465Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:13:43.953465Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:13:43.953465Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:13:43.953465Z", "running": 0, "size": 1}}] 2022-03-10T02:13:54.170 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:13:54.927 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:13:54.927 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:13:54.704694Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:13:54.704694Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:13:54.704694Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:13:54.704694Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:13:54.704694Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:13:54.704694Z", "running": 0, "size": 1}}] 2022-03-10T02:14:04.951 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:14:05.691 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:14:05.692 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:14:05.487258Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:14:05.487258Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:14:05.487258Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:14:05.487258Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:14:05.487258Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:14:05.487258Z", "running": 0, "size": 1}}] 2022-03-10T02:14:15.714 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:14:16.421 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:14:16.422 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:14:16.233778Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:14:16.233778Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:14:16.233778Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:14:16.233778Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:14:16.233778Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:14:16.233778Z", "running": 0, "size": 1}}] 2022-03-10T02:14:26.446 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:14:27.135 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:14:27.136 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:14:26.952672Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:14:26.952672Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:14:26.952672Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:14:26.952672Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:14:26.952672Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:14:26.952672Z", "running": 0, "size": 1}}] 2022-03-10T02:14:37.159 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:14:37.869 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:14:37.869 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:14:37.674631Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:14:37.674631Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:14:37.674631Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:14:37.674631Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:14:37.674631Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:14:37.674631Z", "running": 0, "size": 1}}] 2022-03-10T02:14:47.890 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:14:48.576 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:14:48.577 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:14:48.391151Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:14:48.391151Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:14:48.391151Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:14:48.391151Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:14:48.391151Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:14:48.391151Z", "running": 0, "size": 1}}] 2022-03-10T02:14:58.599 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:14:59.317 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:14:59.319 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:14:59.124381Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:14:59.124381Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:14:59.124381Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:14:59.124381Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:14:59.124381Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:14:59.124381Z", "running": 0, "size": 1}}] 2022-03-10T02:15:09.341 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:15:10.085 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:15:10.085 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:15:09.873376Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:15:09.873376Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:15:09.873376Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:15:09.873376Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:15:09.873376Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:15:09.873376Z", "running": 0, "size": 1}}] 2022-03-10T02:15:20.107 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:15:20.825 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:15:20.825 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:15:20.655150Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:15:20.655150Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:15:20.655150Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:15:20.655150Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:15:20.655150Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:15:20.655150Z", "running": 0, "size": 1}}] 2022-03-10T02:15:30.847 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:15:31.568 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:15:31.569 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:15:31.362178Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:15:31.362178Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:15:31.362178Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:15:31.362178Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:15:31.362178Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:15:31.362178Z", "running": 0, "size": 1}}] 2022-03-10T02:15:41.592 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:15:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:15:42.619 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:15:42.099848Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:15:42.099848Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:15:42.099848Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:15:42.099848Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:15:42.099848Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:15:42.099848Z", "running": 0, "size": 1}}] 2022-03-10T02:15:52.642 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:15:53.364 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:15:53.364 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:15:53.160562Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:15:53.160562Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:15:53.160562Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:15:53.160562Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:15:53.160562Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:15:53.160562Z", "running": 0, "size": 1}}] 2022-03-10T02:16:03.390 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:16:04.132 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:16:04.132 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:16:03.937517Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:16:03.937517Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:16:03.937517Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:16:03.937517Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:16:03.937517Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:16:03.937517Z", "running": 0, "size": 1}}] 2022-03-10T02:16:14.157 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:16:14.856 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:16:14.856 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:16:14.661928Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:16:14.661928Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:16:14.661928Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:16:14.661928Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:16:14.661928Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:16:14.661928Z", "running": 0, "size": 1}}] 2022-03-10T02:16:24.881 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:16:25.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:16:25.578 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:16:25.407975Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:16:25.407975Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:16:25.407975Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:16:25.407975Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:16:25.407975Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:16:25.407975Z", "running": 0, "size": 1}}] 2022-03-10T02:16:35.607 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:16:36.352 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:16:36.353 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:16:36.147819Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:16:36.147819Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:16:36.147819Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:16:36.147819Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:16:36.147819Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:16:36.147819Z", "running": 0, "size": 1}}] 2022-03-10T02:16:46.377 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:16:47.077 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:16:47.078 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:16:46.879249Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:16:46.879249Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:16:46.879249Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:16:46.879249Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:16:46.879249Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:16:46.879249Z", "running": 0, "size": 1}}] 2022-03-10T02:16:57.102 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:16:57.784 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:16:57.784 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:16:57.605710Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:16:57.605710Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:16:57.605710Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:16:57.605710Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:16:57.605710Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:16:57.605710Z", "running": 0, "size": 1}}] 2022-03-10T02:17:07.803 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:17:08.552 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:17:08.552 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:17:08.323503Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:17:08.323503Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:17:08.323503Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:17:08.323503Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:17:08.323503Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:17:08.323503Z", "running": 0, "size": 1}}] 2022-03-10T02:17:18.575 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:17:19.340 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:17:19.341 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:17:19.139375Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:17:19.139375Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:17:19.139375Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:17:19.139375Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:17:19.139375Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:17:19.139375Z", "running": 0, "size": 1}}] 2022-03-10T02:17:29.362 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:17:30.056 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:17:30.057 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:17:29.884702Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:17:29.884702Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:17:29.884702Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:17:29.884702Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:17:29.884702Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:17:29.884702Z", "running": 0, "size": 1}}] 2022-03-10T02:17:40.079 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:17:40.942 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:17:40.942 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:17:40.708794Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:17:40.708794Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:17:40.708794Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:17:40.708794Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:17:40.708794Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:17:40.708794Z", "running": 0, "size": 1}}] 2022-03-10T02:17:50.967 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:17:51.640 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:17:51.641 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:17:51.469260Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:17:51.469260Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:17:51.469260Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:17:51.469260Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:17:51.469260Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:17:51.469260Z", "running": 0, "size": 1}}] 2022-03-10T02:18:01.662 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-lc2pk -- ceph orch ls -f json 2022-03-10T02:18:02.375 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-10T02:18:02.376 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:58.000000Z", "last_refresh": "2022-03-10T02:18:02.188014Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:56:44.000000Z", "last_refresh": "2022-03-10T02:18:02.188014Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:54:20.000000Z", "last_refresh": "2022-03-10T02:18:02.188014Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "4b871cde5e1a0ee955665868a9192070abb1f1a64b0e05f651726af4103cdfda", "container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "created": "2022-03-10T01:58:22.000000Z", "last_refresh": "2022-03-10T02:18:02.188014Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-10T02:18:02.188014Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:7a44e9ebca556862e9704be295a07719d7db47b3", "last_refresh": "2022-03-10T02:18:02.188014Z", "running": 0, "size": 1}}] 2022-03-10T02:18:02.400 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:18:02.401 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:18:02.402 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-10T02:18:02.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T02:18:02.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T02:18:02.432 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-10T02:18:02.448 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:18:02.449 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:18:02.462 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-10T02:18:02.477 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-10T02:18:02.547 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-10T02:18:02.555 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_7a44e9ebca556862e9704be295a07719d7db47b3/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_ff5db9169566776b2f41faf671316ca58a318ea6/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-10T02:18:02.556 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete -f cluster.yaml 2022-03-10T02:18:02.706 INFO:teuthology.orchestra.run.smithi007.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-10T02:18:02.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.701786 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-10T02:18:02.715 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.701975 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-10T02:18:02.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.702361 I | operator: successfully started the controller-runtime manager 2022-03-10T02:18:02.719 INFO:tasks.rook.operator.smithi007.stdout:I0310 02:18:02.702458 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-10T02:18:02.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.708313 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-10T02:18:02.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.708339 I | operator: watching all namespaces for Ceph CRs 2022-03-10T02:18:02.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.708375 I | operator: setting up schemes 2022-03-10T02:18:02.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.711728 I | operator: setting up the controller-runtime manager 2022-03-10T02:18:02.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.915938 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-10T02:18:02.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.918880 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-10T02:18:02.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.918922 I | ceph-cluster-controller: successfully started 2022-03-10T02:18:02.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.918986 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-10T02:18:02.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919011 I | ceph-crashcollector-controller: successfully started 2022-03-10T02:18:02.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919046 I | ceph-block-pool-controller: successfully started 2022-03-10T02:18:02.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919070 I | ceph-object-store-user-controller: successfully started 2022-03-10T02:18:02.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919098 I | ceph-object-realm-controller: successfully started 2022-03-10T02:18:02.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919122 I | ceph-object-zonegroup-controller: successfully started 2022-03-10T02:18:02.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919137 I | ceph-object-zone-controller: successfully started 2022-03-10T02:18:02.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919307 I | ceph-object-controller: successfully started 2022-03-10T02:18:02.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919355 I | ceph-file-controller: successfully started 2022-03-10T02:18:02.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919396 I | ceph-nfs-controller: successfully started 2022-03-10T02:18:02.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919430 I | ceph-rbd-mirror-controller: successfully started 2022-03-10T02:18:02.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919461 I | ceph-client-controller: successfully started 2022-03-10T02:18:02.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919484 I | ceph-filesystem-mirror-controller: successfully started 2022-03-10T02:18:02.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919521 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-10T02:18:02.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919546 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-10T02:18:02.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919565 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-10T02:18:02.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919590 I | ceph-bucket-topic: successfully started 2022-03-10T02:18:02.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919611 I | ceph-bucket-notification: successfully started 2022-03-10T02:18:02.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919623 I | ceph-bucket-notification: successfully started 2022-03-10T02:18:02.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.919637 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-10T02:18:02.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:02.920880 I | operator: starting the controller-runtime manager 2022-03-10T02:18:03.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.025959 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-10T02:18:03.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.026021 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-10T02:18:03.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.026043 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-10T02:18:03.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.030423 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-10T02:18:03.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.032374 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:03.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.086199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:03.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.100127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:03.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.720659 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:03.710+0000 7fe23a8c3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:03.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:03.710+0000 7fe23a8c3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:03.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:03.710+0000 7fe23a8c3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:03.733 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:03.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:03.750385 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:04.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:04.466976 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:04.454+0000 7f496c98a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:04.480 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:04.454+0000 7f496c98a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:04.480 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:04.454+0000 7f496c98a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:04.480 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:04.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:04.485605 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:05.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:05.162511 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:05.150+0000 7f8a4bc4d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:05.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:05.150+0000 7f8a4bc4d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:05.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:05.150+0000 7f8a4bc4d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:05.175 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:05.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:05.189073 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:05.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:05.832776 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:05.822+0000 7f7061caa380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:05.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:05.822+0000 7f7061caa380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:05.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:05.822+0000 7f7061caa380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:05.845 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:05.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:05.878546 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:06.526 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:06.514289 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:06.502+0000 7f37dba0e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:06.527 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:06.502+0000 7f37dba0e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:06.527 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:06.502+0000 7f37dba0e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:06.527 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:06.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:06.649767 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:07.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:07.437016 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:07.426+0000 7f83aec02380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:07.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:07.426+0000 7f83aec02380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:07.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:07.426+0000 7f83aec02380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:07.450 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:07.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:07.604853 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:08.292 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:08.280216 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:08.270+0000 7f032366d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:08.292 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:08.270+0000 7f032366d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:08.293 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:08.270+0000 7f032366d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:08.293 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:08.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:08.606912 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:09.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:09.251400 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:09.238+0000 7f0380288380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:09.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:09.238+0000 7f0380288380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:09.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:09.238+0000 7f0380288380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:09.264 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:09.913 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:09.902018 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:10.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:10.619094 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:10.606+0000 7f601cb5f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:10.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:10.606+0000 7f601cb5f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:10.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:10.606+0000 7f601cb5f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:10.632 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:11.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:11.908314 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:12.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:12.546914 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:12.534+0000 7f835e32b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:12.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:12.534+0000 7f835e32b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:12.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:12.534+0000 7f835e32b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:12.560 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:13.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:13.301522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:13.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:13.313018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:15.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:15.120120 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:15.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:15.803302 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:15.794+0000 7f2f25901380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:15.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:15.794+0000 7f2f25901380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:15.816 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:15.794+0000 7f2f25901380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:15.816 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:20.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:20.929513 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:21.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:21.569366 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:21.558+0000 7f22bd8ad380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:21.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:21.558+0000 7f22bd8ad380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:21.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:21.558+0000 7f22bd8ad380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:21.582 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:23.399 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:23.388043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:23.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:23.398748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:31.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:31.821504 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:32.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:32.468042 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:32.458+0000 7f41c9c43380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:32.480 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:32.458+0000 7f41c9c43380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:32.480 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:32.458+0000 7f41c9c43380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:32.481 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:33.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:33.463294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:33.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:33.477812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:43.720 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:43.708338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:43.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:43.723722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:52.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:52.956832 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:18:53.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:53.589274 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:18:53.578+0000 7f628756d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:18:53.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:53.578+0000 7f628756d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:18:53.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:18:53.578+0000 7f628756d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:18:53.601 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:18:53.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:53.834637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:18:53.857 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:18:53.846107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:03.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:03.921901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:03.946 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:03.935004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:14.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:14.067502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:14.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:14.108545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:24.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:24.212943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:24.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:24.226664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:34.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:34.290335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:34.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:34.302635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:34.568 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:34.557574 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:19:35.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:35.210027 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:19:35.198+0000 7f7ae9b57380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:19:35.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:19:35.198+0000 7f7ae9b57380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:19:35.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:19:35.198+0000 7f7ae9b57380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:19:35.222 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:19:44.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:44.418081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:44.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:44.429709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:54.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:54.530953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:19:54.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:19:54.544708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:04.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:04.614925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:04.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:04.626367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:14.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:14.731083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:14.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:14.744240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:24.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:24.854625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:24.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:24.869204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:34.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:34.968457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:35.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:35.039816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:45.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:45.142509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:45.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:45.158003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:55.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:55.273800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:55.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:55.287518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:20:57.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:57.137891 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:20:57.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:20:57.815953 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:20:57.806+0000 7fbc202a2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:20:57.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:20:57.806+0000 7fbc202a2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:20:57.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:20:57.806+0000 7fbc202a2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:20:57.826 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:21:05.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:05.358141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:05.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:05.407892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:15.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:15.479245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:15.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:15.528838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:25.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:25.675580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:25.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:25.688611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:35.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:35.854400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:35.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:35.866815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:45.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:45.947755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:45.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:45.991606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:56.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:56.125559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:21:56.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:21:56.138770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:06.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:06.250581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:06.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:06.265128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:16.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:16.354740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:16.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:16.368383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:26.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:26.506130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:26.527 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:26.519711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:36.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:36.698344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:36.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:36.709370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:46.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:46.776255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:46.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:46.789822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:56.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:56.860647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:22:56.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:22:56.873645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:06.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:06.936341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:07.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:06.996664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:17.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:17.094922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:17.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:17.106979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:27.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:27.168798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:27.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:27.219156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:37.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:37.309437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:37.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:37.324394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:41.673 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:41.665593 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:23:42.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:42.318775 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:23:42.305+0000 7fa9094e9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:23:42.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:23:42.305+0000 7fa9094e9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:23:42.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:23:42.305+0000 7fa9094e9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:23:42.327 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:23:47.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:47.401613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:47.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:47.488044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:57.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:57.572326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:23:57.591 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:23:57.584808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:07.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:07.653793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:07.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:07.683501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:17.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:17.751145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:17.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:17.761574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:27.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:27.847943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:27.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:27.891846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:37.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:37.956082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:37.972 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:37.967089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:48.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:48.102186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:48.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:48.117144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:58.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:58.190868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:24:58.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:24:58.203380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:08.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:08.313734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:08.332 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:08.327228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:18.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:18.439166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:18.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:18.451144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:28.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:28.548306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:28.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:28.560834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:38.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:38.661775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:38.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:38.673286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:48.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:48.779935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:48.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:48.796421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:58.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:58.873179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:25:58.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:25:58.923061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:09.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:09.042585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:09.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:09.059550 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:19.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:19.270666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:19.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:19.279163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:29.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:29.351732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:29.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:29.413319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:39.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:39.515187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:39.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:39.528625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:49.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:49.607052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:49.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:49.620814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:59.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:59.701642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:26:59.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:26:59.714657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:09.781 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:09.779032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:09.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:09.794371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:19.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:19.884267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:19.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:19.898083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:30.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:30.018321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:30.034 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:30.032532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:40.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:40.107816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:40.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:40.121965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:50.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:50.237410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:27:50.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:27:50.252708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:00.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:00.346789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:00.360 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:00.359373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:10.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:10.543733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:10.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:10.562523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:20.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:20.758150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:20.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:20.768656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:30.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:30.874611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:30.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:30.889137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:40.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:40.966148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:40.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:40.981561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:51.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:51.051332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:28:51.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:28:51.063270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:01.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:01.167716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:01.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:01.182116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:10.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:10.012130 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:29:10.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:10.774078 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:29:10.763+0000 7fe9aef49380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:29:10.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:29:10.763+0000 7fe9aef49380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:29:10.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:29:10.763+0000 7fe9aef49380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:29:10.775 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:29:11.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:11.280303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:11.293 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:11.293794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:21.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:21.405193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:21.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:21.420358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:31.511 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:31.511188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:31.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:31.525686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:41.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:41.672031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:41.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:41.684776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:51.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:51.801019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:29:51.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:29:51.814106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:01.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:01.927765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:01.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:01.941248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:12.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:12.047059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:12.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:12.061147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:22.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:22.151900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:22.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:22.166285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:32.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:32.261909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:32.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:32.279704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:42.377 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:42.358264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:42.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:42.397815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:52.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:52.542247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:30:52.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:30:52.554058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:02.673 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:02.674715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:02.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:02.688125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:12.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:12.763799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:12.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:12.778170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:22.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:22.859924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:22.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:22.916184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:33.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:33.030303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:33.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:33.039867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:43.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:43.101534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:43.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:43.115794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:53.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:53.215857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:31:53.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:31:53.228594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:03.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:03.357970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:03.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:03.371602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:13.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:13.439465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:13.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:13.453611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:23.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:23.558241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:23.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:23.572107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:33.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:33.738987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:33.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:33.752582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:43.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:43.824407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:43.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:43.836156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:53.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:53.930578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:32:54.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:32:54.035315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:04.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:04.135255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:04.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:04.147287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:14.218 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:14.221099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:14.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:14.286564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:24.395 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:24.397579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:24.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:24.410382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:34.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:34.534163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:34.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:34.551121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:44.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:44.648269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:44.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:44.658725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:54.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:54.756569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:33:54.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:33:54.770684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:04.865 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:04.868245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:04.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:04.880541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:14.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:14.997850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:15.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:15.009494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:25.100 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:25.104279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:25.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:25.118873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:35.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:35.225886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:35.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:35.239136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:45.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:45.348784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:45.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:45.363417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:55.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:55.442187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:34:55.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:34:55.454977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:05.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:05.598037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:05.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:05.614039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:15.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:15.720816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:15.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:15.735684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:25.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:25.811461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:25.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:25.825423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:35.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:35.911651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:35.921 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:35.925708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:45.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:45.997059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:46.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:46.012348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:56.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:56.080920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:35:56.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:35:56.094539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:06.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:06.203641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:06.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:06.216443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:16.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:16.359193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:16.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:16.373144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:26.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:26.475155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:26.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:26.487460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:36.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:36.616482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:36.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:36.632474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:46.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:46.753930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:46.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:46.763009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:56.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:56.865324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:36:56.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:36:56.880254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:07.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:07.006037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:07.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:07.022652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:17.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:17.098439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:17.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:17.111727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:27.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:27.197168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:27.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:27.214870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:37.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:37.332950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:37.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:37.344990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:47.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:47.498817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:47.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:47.511269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:57.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:57.621465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:37:57.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:37:57.637492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:07.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:07.757689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:07.765 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:07.770926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:17.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:17.846676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:17.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:17.861972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:27.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:27.960115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:27.969 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:27.974423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:38.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:38.088084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:38.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:38.103015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:48.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:48.202600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:48.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:48.219534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:58.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:58.356693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:38:58.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:38:58.371780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:08.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:08.496205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:08.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:08.509768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:18.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:18.587725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:18.592 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:18.598306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:28.669 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:28.674578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:28.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:28.686024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:38.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:38.837977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:38.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:38.850879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:48.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:48.964223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:48.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:48.977307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:59.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:59.124824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:39:59.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:39:59.135193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:06.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:06.142048 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:40:06.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:06.802029 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:40:06.789+0000 7fcca117a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:40:06.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:40:06.789+0000 7fcca117a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:40:06.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:40:06.789+0000 7fcca117a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:40:06.798 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:40:09.229 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:09.234092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:09.245 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:09.251688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:19.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:19.360332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:19.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:19.373159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:29.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:29.499524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:29.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:29.514631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:39.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:39.629152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:39.638 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:39.643065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:49.758 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:49.763714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:49.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:49.775081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:59.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:59.846121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:40:59.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:40:59.859086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:09.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:09.967547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:09.974 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:09.980323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:20.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:20.133698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:20.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:20.147467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:30.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:30.285864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:30.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:30.300270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:40.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:40.406376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:40.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:40.421956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:50.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:50.493688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:41:50.502 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:41:50.507722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:00.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:00.579293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:00.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:00.592816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:10.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:10.677494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:10.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:10.716692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:20.809 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:20.815007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:20.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:20.825353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:30.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:30.930300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:30.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:30.943514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:41.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:41.015737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:41.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:41.027751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:51.084 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:51.089552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:42:51.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:42:51.104475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:01.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:01.167338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:01.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:01.204858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:11.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:11.321679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:11.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:11.335760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:21.400 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:21.405495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:21.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:21.417614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:31.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:31.497639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:31.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:31.510803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:41.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:41.616932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:41.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:41.630640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:51.773 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:51.778574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:43:51.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:43:51.790866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:01.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:01.863958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:01.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:01.905695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:11.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:11.973031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:11.981 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:11.987444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:22.094 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:22.100043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:22.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:22.115652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:32.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:32.225072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:32.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:32.238277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:42.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:42.351118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:42.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:42.365542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:52.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:52.448835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:44:52.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:44:52.460666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:02.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:02.568090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:02.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:02.580436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:12.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:12.660944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:12.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:12.672310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:22.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:22.790234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:22.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:22.808820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:32.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:32.903215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:32.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:32.916461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:43.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:43.022822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:43.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:43.035973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:53.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:53.107629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:45:53.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:45:53.121002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:03.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:03.247217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:03.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:03.261489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:13.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:13.341697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:13.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:13.393132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:23.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:23.475098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:23.484 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:23.490324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:33.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:33.628775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:33.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:33.641791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:43.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:43.979016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:43.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:43.992993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:54.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:54.092247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:46:54.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:46:54.105182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:04.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:04.219765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:04.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:04.231732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:14.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:14.363029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:14.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:14.389782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:24.500 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:24.506031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:24.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:24.518232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:34.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:34.605740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:34.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:34.620192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:44.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:44.701183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:44.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:44.712504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:54.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:54.835871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:47:54.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:47:54.849142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:04.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:04.916744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:04.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:04.930067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:14.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:15.001000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:15.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:15.101036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:25.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:25.203480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:25.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:25.218614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:35.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:35.317737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:35.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:35.331354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:45.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:45.415368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:45.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:45.428518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:55.504 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:55.509620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:48:55.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:48:55.586115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:05.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:05.657459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:05.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:05.670795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:15.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:15.784815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:15.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:15.799722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:25.926 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:25.931562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:25.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:25.943445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:36.050 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:36.055983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:36.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:36.071005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:46.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:46.210858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:46.218 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:46.223590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:56.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:56.344306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:49:56.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:49:56.358950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:06.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:06.423955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:06.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:06.471938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:16.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:16.560108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:16.574 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:16.575568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:26.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:26.687705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:26.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:26.702152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:36.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:36.825732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:36.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:36.839039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:46.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:46.910777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:46.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:46.922301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:57.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:57.088264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:50:57.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:50:57.101803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:07.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:07.203220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:07.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:07.218158 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:17.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:17.280617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:17.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:17.346176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:27.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:27.450125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:27.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:27.463239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:37.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:37.570362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:37.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:37.586187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:47.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:47.716397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:47.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:47.729707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:57.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:57.830136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:51:57.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:51:57.844967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:07.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:07.945892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:07.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:07.958144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:18.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:18.030406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:18.035 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:18.041165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:28.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:28.128395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:28.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:28.139912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:38.233 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:38.239836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:38.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:38.253950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:48.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:48.326240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:48.335 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:48.340677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:58.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:58.449981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:52:58.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:52:58.462278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:08.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:08.562590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:08.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:08.575969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:18.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:18.692670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:18.700 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:18.706804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:28.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:28.866369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:28.872 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:28.878080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:38.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:38.947611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:38.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:38.960506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:49.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:49.092721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:49.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:49.125084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:59.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:59.229145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:53:59.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:53:59.244001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:09.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:09.332213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:09.337 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:09.344126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:19.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:19.592683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:19.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:19.607604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:30.011 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:30.017870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:30.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:30.028094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:40.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:40.144831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:40.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:40.158568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:50.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:50.321563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:54:50.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:54:50.335236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:00.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:00.739162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:00.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:00.752242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:10.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:10.818096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:10.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:10.830067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:21.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:21.094259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:21.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:21.114406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:31.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:31.326566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:31.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:31.340346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:41.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:41.415843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:41.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:41.469088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:51.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:51.743497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:55:51.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:55:51.757867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:02.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:02.012981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:02.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:02.026960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:12.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:12.096752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:12.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:12.109655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:22.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:22.207001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:22.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:22.220991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:32.366 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:32.372971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:32.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:32.385285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:42.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:42.456682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:42.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:42.471472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:46.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:46.811881 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T02:56:47.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:47.474384 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T02:56:47.461+0000 7f51b42e7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T02:56:47.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:56:47.461+0000 7f51b42e7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T02:56:47.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T02:56:47.461+0000 7f51b42e7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T02:56:47.469 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T02:56:52.577 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:52.583226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:56:52.592 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:56:52.599583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:02.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:02.714818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:02.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:02.729188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:12.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:12.824607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:12.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:12.837924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:22.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:22.948182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:22.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:22.960641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:33.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:33.083561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:33.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:33.096556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:43.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:43.207433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:43.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:43.220758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:53.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:53.277030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:57:53.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:57:53.287153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:03.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:03.427549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:03.432 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:03.439460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:13.577 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:13.583980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:13.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:13.597400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:23.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:23.707567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:23.715 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:23.721734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:33.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:33.814129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:33.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:33.832766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:43.921 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:43.927581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:43.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:43.940131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:54.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:54.082949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:58:54.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:58:54.096400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:04.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:04.172287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:04.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:04.185253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:14.302 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:14.307678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:14.314 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:14.319864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:24.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:24.431841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:24.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:24.445552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:34.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:34.527830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:34.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:34.561075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:44.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:44.651899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:44.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:44.664373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:54.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:54.804738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T02:59:54.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 02:59:54.822221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:04.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:04.885709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:04.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:04.921159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:14.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:14.992444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:15.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:15.005107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:25.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:25.094662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:25.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:25.106453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:35.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:35.302037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:35.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:35.314223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:45.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:45.434429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:45.448 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:45.451604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:55.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:55.551397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:00:55.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:00:55.579129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:05.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:05.687642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:05.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:05.702172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:15.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:15.779519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:15.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:15.849172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:25.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:25.969656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:25.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:25.983711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:36.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:36.147684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:36.160 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:36.162899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:46.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:46.305184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:46.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:46.320166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:56.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:56.427400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:01:56.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:01:56.440269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:06.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:06.544258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:06.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:06.559295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:16.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:16.669814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:16.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:16.685097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:26.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:26.784706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:26.795 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:26.797323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:36.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:36.954547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:36.964 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:36.965736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:47.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:47.108296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:47.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:47.120116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:57.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:57.277068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:02:57.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:02:57.290005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:07.432 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:07.433723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:07.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:07.446152 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:17.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:17.526526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:17.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:17.540149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:27.638 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:27.638664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:27.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:27.654218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:37.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:37.762170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:37.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:37.775534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:47.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:47.879571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:47.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:47.941882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:58.050 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:58.050575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:03:58.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:03:58.063812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:08.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:08.176009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:08.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:08.191292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:18.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:18.265723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:18.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:18.280823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:28.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:28.396105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:28.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:28.410874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:38.504 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:38.504512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:38.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:38.515278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:48.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:48.587290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:48.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:48.661824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:58.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:58.743217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:04:58.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:04:58.755952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:08.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:08.869707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:08.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:08.878923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:19.050 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:19.049624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:19.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:19.062655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:29.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:29.137645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:29.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:29.149078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:39.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:39.215738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:39.229 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:39.228313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:49.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:49.316588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:49.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:49.328523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:59.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:59.405568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:05:59.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:05:59.417530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:09.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:09.479981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:09.495 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:09.493990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:19.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:19.606664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:19.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:19.619702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:29.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:29.685776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:29.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:29.699355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:39.822 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:39.820130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:39.836 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:39.834871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:49.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:49.935480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:06:49.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:06:49.946536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:00.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:00.025707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:00.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:00.073905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:10.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:10.183802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:10.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:10.197566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:20.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:20.321558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:20.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:20.336306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:30.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:30.404751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:30.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:30.418047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:40.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:40.641222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:40.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:40.655233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:50.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:50.789595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:07:50.808 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:07:50.805105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:00.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:00.936969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:00.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:00.950951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:11.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:11.027098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:11.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:11.039175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:21.155 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:21.152463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:21.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:21.165614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:31.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:31.233079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:31.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:31.246130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:41.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:41.398237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:41.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:41.412576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:51.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:51.517454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:08:51.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:08:51.531856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:01.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:01.639863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:01.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:01.653054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:11.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:11.765694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:11.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:11.778733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:21.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:21.849718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:21.865 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:21.862564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:31.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:31.967753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:31.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:31.984134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:42.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:42.051803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:42.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:42.108434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:52.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:52.224794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:09:52.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:09:52.238881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:02.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:02.369240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:02.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:02.384487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:12.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:12.454283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:12.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:12.526667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:22.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:22.651100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:22.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:22.664840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:32.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:32.785564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:32.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:32.800891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:42.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:42.897020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:42.913 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:42.910853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:53.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:53.010505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:10:53.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:10:53.025079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:03.103 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:03.100675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:03.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:03.114323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:13.230 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:13.227167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:13.244 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:13.242151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:23.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:23.322432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:23.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:23.336525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:33.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:33.485167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:33.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:33.500601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:43.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:43.626288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:43.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:43.638358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:53.720 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:53.717368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:11:53.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:11:53.769245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:03.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:03.914178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:04.334 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:03.927331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:14.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:14.043947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:14.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:14.073318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:24.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:24.163020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:24.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:24.177849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:34.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:34.282711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:34.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:34.296625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:44.463 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:44.460049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:44.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:44.474587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:54.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:54.573216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:12:54.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:12:54.587514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:04.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:04.663911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:04.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:04.679217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:14.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:14.783314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:14.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:14.795821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:24.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:24.907251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:24.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:24.922169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:27.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:27.485653 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T03:13:28.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:28.140512 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:13:28.129+0000 7fb4ba7ce380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:13:28.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T03:13:28.129+0000 7fb4ba7ce380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:13:28.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T03:13:28.129+0000 7fb4ba7ce380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:13:28.145 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:13:35.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:35.035269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:35.050 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:35.048420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:45.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:45.144366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:45.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:45.205015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:55.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:55.338834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:13:55.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:13:55.352931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:05.433 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:05.431664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:05.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:05.480512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:15.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:15.547748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:15.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:15.561768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:25.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:25.684983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:25.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:25.696994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:35.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:35.769433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:35.836 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:35.834582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:45.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:45.974768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:45.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:45.988909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:56.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:56.098986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:14:56.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:14:56.111460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:06.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:06.191654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:06.252 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:06.250327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:16.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:16.410565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:16.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:16.424259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:26.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:26.526595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:26.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:26.541167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:36.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:36.602415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:36.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:36.665540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:46.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:46.798447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:46.809 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:46.807929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:56.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:56.888482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:15:56.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:15:56.900345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:07.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:07.069502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:07.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:07.087522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:17.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:17.168689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:17.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:17.183646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:27.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:27.284983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:27.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:27.297371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:37.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:37.390218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:37.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:37.410918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:47.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:47.505380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:47.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:47.516586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:57.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:57.624565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:16:57.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:16:57.655474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:07.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:07.731765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:07.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:07.745009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:17.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:17.861379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:17.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:17.875071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:28.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:28.060292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:28.074 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:28.073785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:38.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:38.131692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:38.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:38.144686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:48.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:48.250397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:48.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:48.264605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:58.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:58.453530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:17:58.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:17:58.467260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:08.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:08.551998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:08.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:08.564210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:18.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:18.641043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:18.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:18.692532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:28.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:28.811934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:28.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:28.834494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:39.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:39.086077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:39.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:39.098283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:49.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:49.175947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:49.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:49.189169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:59.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:59.279548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:18:59.295 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:18:59.294458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:09.360 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:09.359015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:09.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:09.370398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:19.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:19.444481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:19.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:19.458755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:29.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:29.555195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:29.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:29.568566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:39.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:39.624026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:39.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:39.641524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:49.715 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:49.714517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:49.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:49.728482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:59.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:59.799519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:19:59.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:19:59.811954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:09.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:09.914636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:09.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:09.929501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:20.064 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:20.063510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:20.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:20.078018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:30.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:30.192217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:30.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:30.207534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:40.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:40.305551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:40.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:40.317893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:50.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:50.408632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:20:50.421 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:20:50.420448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:00.516 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:00.515861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:00.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:00.529818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:10.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:10.605950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:10.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:10.617542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:20.691 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:20.690908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:20.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:20.750345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:30.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:30.868344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:30.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:30.881130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:40.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:40.987161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:41.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:41.004813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:51.065 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:51.064243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:21:51.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:21:51.079114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:01.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:01.182142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:01.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:01.193743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:11.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:11.257449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:11.271 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:11.271189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:21.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:21.339718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:21.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:21.354780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:31.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:31.476983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:31.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:31.492902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:41.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:41.627855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:41.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:41.643709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:51.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:51.749636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:22:51.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:22:51.763278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:01.834 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:01.833611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:01.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:01.848374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:11.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:11.932391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:11.946 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:11.946116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:22.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:22.067961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:22.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:22.082941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:32.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:32.149113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:32.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:32.165067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:42.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:42.254699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:42.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:42.270064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:52.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:52.368987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:23:52.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:23:52.381914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:02.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:02.495425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:02.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:02.508853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:12.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:12.611225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:12.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:12.625318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:22.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:22.698752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:22.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:22.711197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:32.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:32.813374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:32.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:32.825335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:42.972 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:42.972135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:42.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:42.985713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:53.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:53.056681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:24:53.068 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:24:53.067605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:03.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:03.150245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:03.163 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:03.162914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:13.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:13.236281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:13.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:13.252225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:23.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:23.319869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:23.377 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:23.376167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:33.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:33.470587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:33.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:33.533208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:43.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:43.605535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:43.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:43.659146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:53.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:53.767278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:25:53.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:25:53.782449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:03.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:03.904298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:03.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:03.916543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:13.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:13.983765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:13.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:13.998189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:24.141 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:24.140302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:24.155 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:24.154683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:34.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:34.296864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:34.308 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:34.308424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:44.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:44.410328 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:44.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:44.425695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:54.495 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:54.492333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:26:54.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:26:54.507706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:04.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:04.566947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:04.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:04.580265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:14.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:14.658402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:14.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:14.671254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:24.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:24.734968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:24.791 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:24.791083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:34.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:34.893976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:34.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:34.909382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:45.034 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:45.033776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:45.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:45.052219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:55.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:55.139372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:27:55.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:27:55.206869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:05.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:05.274980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:05.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:05.310725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:15.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:15.397697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:15.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:15.412764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:25.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:25.495979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:25.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:25.540233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:35.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:35.640160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:35.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:35.655109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:45.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:45.734069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:45.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:45.782275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:55.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:55.852421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:28:55.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:28:55.868008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:05.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:05.985465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:06.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:06.001804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:16.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:16.076576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:16.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:16.134237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:26.201 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:26.201875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:26.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:26.215405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:36.399 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:36.399400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:36.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:36.415181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:46.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:46.496798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:46.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:46.542754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:56.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:56.621631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:29:56.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:29:56.633144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:06.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:06.695226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:06.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:06.707646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:08.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:08.148422 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T03:30:08.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:08.784112 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:30:08.777+0000 7f70c5aaf380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:30:08.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T03:30:08.777+0000 7f70c5aaf380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:30:08.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T03:30:08.777+0000 7f70c5aaf380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:30:08.785 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:30:16.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:16.779274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:16.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:16.794375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:26.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:26.891009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:26.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:26.902886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:37.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:37.005523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:37.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:37.017467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:47.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:47.134335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:47.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:47.147960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:57.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:57.210493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:30:57.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:30:57.323486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:07.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:07.422872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:07.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:07.487198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:17.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:17.562727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:17.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:17.576842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:27.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:27.687806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:27.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:27.699677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:37.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:37.770182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:37.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:37.782723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:47.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:47.985139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:47.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:47.999159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:58.233 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:58.233523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:31:58.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:31:58.249350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:08.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:08.307807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:08.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:08.323655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:18.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:18.427767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:18.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:18.443700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:28.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:28.538518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:28.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:28.552233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:38.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:38.665776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:38.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:38.678246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:48.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:48.772937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:48.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:48.787872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:58.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:58.878917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:32:58.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:32:58.895689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:09.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:09.026204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:09.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:09.040430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:19.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:19.167737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:19.181 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:19.181777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:29.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:29.250394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:29.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:29.260069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:39.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:39.330967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:39.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:39.344187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:49.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:49.450878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:49.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:49.465472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:59.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:59.536112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:33:59.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:33:59.549237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:09.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:09.630517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:09.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:09.641409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:19.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:19.750643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:19.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:19.762463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:29.864 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:29.864075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:29.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:29.876058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:39.981 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:39.981015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:39.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:39.993778 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:50.103 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:50.102497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:34:50.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:34:50.115997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:00.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:00.223544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:00.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:00.236744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:10.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:10.307061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:10.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:10.319979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:20.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:20.438572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:20.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:20.452471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:30.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:30.589136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:30.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:30.604689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:40.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:40.708657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:40.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:40.720815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:50.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:50.777127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:35:50.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:35:50.791340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:00.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:00.890314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:00.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:00.903685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:11.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:11.001287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:11.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:11.015235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:21.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:21.103276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:21.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:21.115526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:31.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:31.210961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:31.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:31.290436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:41.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:41.361156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:41.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:41.371493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:51.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:51.514501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:36:51.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:36:51.530346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:01.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:01.599347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:01.615 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:01.613795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:11.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:11.714899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:11.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:11.727771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:21.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:21.815710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:21.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:21.827447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:31.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:31.960561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:31.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:31.972720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:42.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:42.049892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:42.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:42.108759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:52.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:52.211240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:37:52.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:37:52.224417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:02.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:02.342358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:02.360 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:02.359482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:12.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:12.469704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:12.483 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:12.482245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:22.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:22.557870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:22.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:22.571568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:32.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:32.680486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:32.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:32.695421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:42.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:42.769710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:42.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:42.879318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:52.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:52.937262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:38:52.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:38:52.951784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:03.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:03.010986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:03.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:03.021462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:13.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:13.123389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:13.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:13.136656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:23.205 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:23.203890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:23.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:23.218935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:33.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:33.296750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:33.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:33.309486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:43.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:43.411477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:43.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:43.423421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:53.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:53.494176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:39:53.511 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:39:53.509177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:03.619 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:03.616856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:03.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:03.629781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:13.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:13.765104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:13.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:13.778735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:23.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:23.928083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:23.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:23.941897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:34.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:34.018815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:34.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:34.032117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:44.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:44.121183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:44.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:44.132210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:54.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:54.209640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:40:54.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:40:54.224228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:04.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:04.329098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:04.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:04.343130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:14.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:14.447698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:14.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:14.466006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:24.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:24.575733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:24.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:24.591092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:34.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:34.699307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:34.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:34.714651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:44.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:44.843839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:44.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:44.856728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:54.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:54.977907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:41:54.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:41:54.990320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:05.124 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:05.121469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:05.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:05.133811 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:15.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:15.201950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:15.216 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:15.214318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:25.302 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:25.299683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:25.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:25.313169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:35.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:35.417664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:35.433 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:35.430990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:45.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:45.517558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:45.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:45.528007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:55.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:55.629437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:42:55.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:42:55.644538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:05.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:05.714389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:05.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:05.725870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:15.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:15.821573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:15.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:15.834526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:25.906 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:25.903125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:25.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:25.917460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:36.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:36.015472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:36.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:36.029238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:46.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:46.135631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:46.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:46.149351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:56.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:56.212886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:43:56.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:43:56.258988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:06.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:06.352986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:06.370 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:06.367376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:16.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:16.447438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:16.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:16.462489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:26.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:26.531067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:26.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:26.543615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:36.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:36.751994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:36.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:36.765742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:46.832 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:46.829245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:46.913 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:46.909580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:56.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:56.988519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:44:57.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:44:57.048601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:07.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:07.154810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:07.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:07.171076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:17.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:17.236146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:17.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:17.285380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:27.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:27.416416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:27.433 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:27.430250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:37.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:37.548084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:37.566 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:37.562459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:47.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:47.624177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:47.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:47.637989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:57.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:57.739127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:45:57.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:45:57.751397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:07.818 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:07.810763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:07.866 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:07.862796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:17.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:17.978744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:17.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:17.992313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:28.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:28.102397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:28.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:28.116701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:38.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:38.185142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:38.201 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:38.198394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:48.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:48.305364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:48.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:48.319589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:48.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:48.791011 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T03:46:49.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:49.466956 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T03:46:49.456+0000 7f92e8f0a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T03:46:49.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T03:46:49.456+0000 7f92e8f0a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T03:46:49.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T03:46:49.456+0000 7f92e8f0a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T03:46:49.472 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T03:46:58.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:58.456037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:46:58.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:46:58.469933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:08.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:08.560077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:08.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:08.576004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:18.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:18.701908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:18.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:18.714395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:28.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:28.787000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:28.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:28.797615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:38.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:38.914495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:38.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:38.928664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:49.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:49.057504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:49.079 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:49.075648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:59.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:59.184738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:47:59.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:47:59.196941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:09.300 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:09.296406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:09.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:09.309279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:19.381 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:19.377652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:19.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:19.392522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:29.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:29.502296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:29.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:29.517502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:39.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:39.611974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:39.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:39.625724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:49.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:49.736557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:49.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:49.750007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:59.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:59.818916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:48:59.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:48:59.833416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:09.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:09.931877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:09.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:09.946100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:20.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:20.020770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:20.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:20.034242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:30.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:30.124024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:30.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:30.133581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:40.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:40.193567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:40.257 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:40.253497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:50.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:50.318001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:49:50.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:49:50.332156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:00.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:00.419111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:00.483 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:00.478708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:10.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:10.581258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:10.599 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:10.595197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:20.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:20.666229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:20.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:20.679326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:30.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:30.764537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:30.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:30.782676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:40.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:40.876173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:40.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:40.889991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:51.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:51.000833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:50:51.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:50:51.014437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:01.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:01.080725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:01.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:01.094391 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:11.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:11.280231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:11.296 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:11.291918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:21.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:21.514127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:21.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:21.528447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:31.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:31.682631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:31.700 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:31.696365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:41.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:41.929934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:41.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:41.944633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:52.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:52.049720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:51:52.068 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:51:52.063976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:02.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:02.173706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:02.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:02.187388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:12.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:12.304715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:12.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:12.316654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:22.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:22.421436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:22.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:22.436380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:32.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:32.502612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:32.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:32.515421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:42.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:42.663420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:42.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:42.676335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:52.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:52.778137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:52:52.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:52:52.792687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:02.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:02.865107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:02.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:02.878164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:12.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:12.945358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:12.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:12.958263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:23.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:23.023752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:23.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:23.034325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:33.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:33.104741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:33.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:33.117852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:43.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:43.219670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:43.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:43.233537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:53.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:53.334406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:53:53.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:53:53.347640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:03.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:03.444087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:03.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:03.457530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:13.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:13.532352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:13.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:13.544913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:23.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:23.628846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:23.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:23.643068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:33.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:33.770456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:33.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:33.784873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:43.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:43.887746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:43.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:43.900210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:54.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:54.012482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:54:54.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:54:54.025583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:04.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:04.143775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:04.163 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:04.157357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:14.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:14.283032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:14.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:14.296159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:24.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:24.409434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:24.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:24.424345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:34.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:34.516307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:34.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:34.529642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:44.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:44.613106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:44.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:44.625766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:54.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:54.771650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:55:54.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:55:54.783308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:04.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:04.847287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:04.867 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:04.861965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:14.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:14.930898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:14.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:14.943341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:25.016 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:25.010267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:25.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:25.065990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:35.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:35.180542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:35.202 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:35.196395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:45.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:45.263028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:45.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:45.275409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:55.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:55.339742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:56:55.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:56:55.393405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:05.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:05.511253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:05.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:05.524982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:15.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:15.606306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:15.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:15.623299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:25.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:25.720188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:25.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:25.736058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:35.822 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:35.816410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:35.834 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:35.828516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:45.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:45.953616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:45.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:45.965681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:56.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:56.103076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:57:56.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:57:56.114775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:06.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:06.230683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:06.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:06.245180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:16.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:16.347592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:16.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:16.361678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:26.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:26.431971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:26.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:26.445048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:36.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:36.543383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:36.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:36.557242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:46.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:46.657802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:46.676 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:46.670147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:56.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:56.799260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:58:56.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:58:56.812473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:06.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:06.896594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:06.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:06.947704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:17.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:17.053196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:17.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:17.066634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:27.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:27.185345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:27.205 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:27.199417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:37.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:37.279488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:37.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:37.295453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:47.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:47.406985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:47.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:47.423368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:57.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:57.486364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T03:59:57.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 03:59:57.500497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:07.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:07.609697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:07.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:07.627151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:17.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:17.727726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:17.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:17.742426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:27.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:27.818122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:27.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:27.833412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:37.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:37.975734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:37.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:37.986066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:48.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:48.065345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:48.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:48.079303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:58.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:58.219945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:00:58.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:00:58.233213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:08.360 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:08.354218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:08.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:08.366975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:18.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:18.465696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:18.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:18.479704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:28.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:28.569285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:28.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:28.580874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:38.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:38.669331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:38.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:38.683664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:48.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:48.749111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:48.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:48.761567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:58.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:58.840134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:01:58.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:01:58.887111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:09.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:08.999793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:09.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:09.013349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:19.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:19.125215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:19.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:19.137614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:29.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:29.204252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:29.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:29.217497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:39.332 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:39.325036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:39.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:39.338385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:49.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:49.456123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:49.478 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:49.472131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:59.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:59.574205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:02:59.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:02:59.596081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:09.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:09.676468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:09.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:09.693040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:19.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:19.798909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:19.818 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:19.812221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:29.483 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:29.476669 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T04:03:29.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:29.886221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:29.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:29.898878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:30.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:30.173089 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:03:30.154+0000 7f1791135380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:03:30.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T04:03:30.154+0000 7f1791135380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:03:30.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T04:03:30.154+0000 7f1791135380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:03:30.180 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:03:39.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:39.986479 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:40.008 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:40.001695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:50.100 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:50.093544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:03:50.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:03:50.106159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:00.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:00.187991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:00.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:00.200389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:10.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:10.282108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:10.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:10.342646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:20.463 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:20.456440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:20.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:20.471343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:30.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:30.550804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:30.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:30.561699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:40.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:40.658493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:40.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:40.671853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:50.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:50.753240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:04:50.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:04:50.823070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:00.906 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:00.899563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:00.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:00.908110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:11.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:11.012458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:11.035 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:11.028737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:21.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:21.100403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:21.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:21.112443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:31.218 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:31.211610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:31.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:31.225761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:41.300 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:41.294134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:41.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:41.308726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:51.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:51.430865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:05:51.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:05:51.456215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:01.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:01.537084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:01.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:01.549635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:11.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:11.673474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:11.691 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:11.685196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:21.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:21.782878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:21.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:21.796460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:31.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:31.951174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:31.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:31.960814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:42.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:42.051325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:42.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:42.063120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:52.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:52.131581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:06:52.205 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:06:52.199310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:02.277 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:02.270948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:02.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:02.284166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:12.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:12.396318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:12.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:12.409635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:22.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:22.491506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:22.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:22.542889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:32.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:32.613902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:32.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:32.626894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:42.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:42.719224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:42.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:42.732874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:52.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:52.827091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:07:52.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:07:52.886751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:03.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:03.004271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:03.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:03.014479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:13.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:13.105251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:13.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:13.116151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:23.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:23.187662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:23.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:23.200070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:33.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:33.274734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:33.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:33.288032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:43.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:43.363717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:43.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:43.420350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:53.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:53.516405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:08:53.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:08:53.528452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:03.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:03.605406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:03.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:03.619362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:13.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:13.703510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:13.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:13.718626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:23.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:23.790475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:23.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:23.820558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:33.926 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:33.920548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:33.945 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:33.939270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:44.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:44.091028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:44.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:44.102912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:54.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:54.244963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:09:54.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:09:54.260132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:04.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:04.372713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:04.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:04.387789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:14.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:14.454366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:14.475 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:14.470233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:24.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:24.539198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:24.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:24.553216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:34.669 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:34.663302 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:34.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:34.676519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:44.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:44.750701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:44.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:44.765100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:54.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:54.862121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:10:54.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:10:54.874858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:05.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:05.013177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:05.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:05.027779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:15.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:15.150226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:15.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:15.164203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:25.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:25.355449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:25.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:25.367856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:35.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:35.517750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:35.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:35.531573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:45.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:45.609069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:45.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:45.622784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:55.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:55.763291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:11:55.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:11:55.778287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:05.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:05.959664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:05.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:05.979472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:16.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:16.178717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:16.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:16.193841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:26.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:26.318006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:26.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:26.334202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:36.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:36.436130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:36.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:36.450348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:46.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:46.524271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:46.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:46.537052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:56.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:56.621071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:12:56.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:12:56.633721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:06.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:06.720110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:06.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:06.733860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:16.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:16.852780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:16.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:16.865849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:26.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:26.969456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:26.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:26.983309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:37.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:37.107922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:37.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:37.124007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:47.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:47.177762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:47.261 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:47.255884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:57.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:57.324059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:13:57.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:13:57.338536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:07.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:07.411128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:07.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:07.424314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:17.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:17.503656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:17.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:17.550434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:27.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:27.621088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:27.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:27.635325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:37.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:37.735159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:37.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:37.746329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:47.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:47.813600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:47.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:47.826279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:57.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:57.895137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:14:57.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:14:57.907500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:08.016 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:08.011269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:08.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:08.024381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:18.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:18.090651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:18.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:18.102592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:28.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:28.231384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:28.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:28.244335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:38.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:38.390523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:38.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:38.402199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:48.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:48.534802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:48.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:48.547535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:58.638 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:58.628804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:15:58.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:15:58.638602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:08.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:08.723884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:08.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:08.739967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:18.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:18.869418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:18.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:18.883497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:28.981 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:28.974710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:28.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:28.988573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:39.053 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:39.047212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:39.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:39.062206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:49.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:49.138021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:49.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:49.191226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:59.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:59.314501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:16:59.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:16:59.328180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:09.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:09.397269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:09.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:09.480093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:19.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:19.564567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:19.594 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:19.588952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:29.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:29.712214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:29.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:29.726575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:39.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:39.832912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:39.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:39.845938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:49.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:49.951799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:17:49.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:17:49.965402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:00.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:00.076221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:00.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:00.088661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:10.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:10.182442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:10.202 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:10.196805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:20.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:20.286379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:20.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:20.300573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:30.377 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:30.372161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:30.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:30.384933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:40.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:40.456570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:40.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:40.471559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:50.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:50.599363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:18:50.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:18:50.613017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:00.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:00.710845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:00.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:00.725775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:10.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:10.820543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:10.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:10.830543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:20.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:20.911890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:20.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:20.925630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:31.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:31.007544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:31.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:31.066411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:41.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:41.170800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:41.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:41.186520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:51.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:51.282725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:19:51.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:19:51.294644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:01.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:01.405690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:01.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:01.422054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:10.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:10.180750 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T04:20:10.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:10.924187 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:20:10.914+0000 7fa58137c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:20:10.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T04:20:10.914+0000 7fa58137c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:20:10.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T04:20:10.914+0000 7fa58137c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:20:10.931 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:20:11.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:11.529709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:11.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:11.542865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:21.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:21.661628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:21.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:21.674682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:31.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:31.818036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:31.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:31.828351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:41.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:41.936272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:41.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:41.949728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:52.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:52.043686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:20:52.068 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:20:52.063340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:02.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:02.133937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:02.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:02.148139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:12.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:12.247680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:12.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:12.261990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:22.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:22.324459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:22.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:22.339589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:32.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:32.451420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:32.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:32.464956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:42.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:42.567660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:42.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:42.581588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:52.715 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:52.709582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:21:52.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:21:52.723750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:02.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:02.794492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:02.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:02.807432 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:12.919 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:12.914414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:12.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:12.928156 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:23.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:22.997827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:23.014 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:23.009790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:33.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:33.091453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:33.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:33.107240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:43.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:43.187344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:43.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:43.259820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:53.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:53.376707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:22:53.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:22:53.391021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:03.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:03.497824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:03.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:03.512407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:13.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:13.621474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:13.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:13.635707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:23.719 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:23.713694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:23.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:23.728261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:33.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:33.829179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:33.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:33.841050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:43.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:43.957279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:43.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:43.969621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:54.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:54.096441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:23:54.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:23:54.109543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:04.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:04.204562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:04.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:04.217352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:14.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:14.304856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:14.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:14.368374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:24.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:24.448532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:24.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:24.461744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:34.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:34.574088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:34.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:34.587839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:44.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:44.690859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:44.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:44.704428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:54.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:54.778372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:24:54.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:24:54.790647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:04.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:04.907961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:04.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:04.922396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:14.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:14.986121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:15.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:14.998867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:25.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:25.088643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:25.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:25.099760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:35.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:35.173938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:35.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:35.210992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:45.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:45.400327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:45.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:45.412295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:55.558 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:55.552361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:25:55.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:25:55.565068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:05.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:05.630720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:05.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:05.644451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:15.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:15.748074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:15.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:15.763005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:25.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:25.839764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:25.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:25.854555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:35.969 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:35.964090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:35.981 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:35.975874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:46.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:46.069762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:46.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:46.084500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:56.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:56.186138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:26:56.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:26:56.200361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:06.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:06.293721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:06.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:06.305423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:16.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:16.403865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:16.419 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:16.413656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:26.483 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:26.477455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:26.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:26.491759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:36.610 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:36.604159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:36.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:36.618634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:46.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:46.712031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:46.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:46.726501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:56.867 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:56.861400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:27:56.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:27:56.875914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:06.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:06.976283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:07.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:06.994298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:17.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:17.071350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:17.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:17.085362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:27.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:27.152782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:27.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:27.214990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:37.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:37.282008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:37.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:37.293018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:47.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:47.369436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:47.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:47.382880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:57.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:57.558472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:28:57.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:28:57.572904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:07.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:07.645734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:07.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:07.699659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:17.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:17.770471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:17.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:17.783228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:27.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:27.887999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:27.905 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:27.899407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:37.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:37.969802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:37.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:37.980669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:48.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:48.055862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:48.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:48.069554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:58.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:58.151413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:29:58.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:29:58.161061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:08.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:08.258249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:08.277 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:08.271732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:18.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:18.349355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:18.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:18.406956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:28.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:28.549930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:28.572 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:28.566672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:38.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:38.688824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:38.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:38.702459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:48.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:48.769457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:48.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:48.825241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:58.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:58.893661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:30:58.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:30:58.951445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:09.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:09.053983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:09.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:09.068065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:19.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:19.130872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:19.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:19.145421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:29.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:29.240332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:29.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:29.253245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:39.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:39.327096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:39.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:39.341715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:49.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:49.464355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:49.483 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:49.477364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:59.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:59.580711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:31:59.599 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:31:59.593918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:09.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:09.692789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:09.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:09.708134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:19.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:19.778599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:19.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:19.792644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:29.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:29.885631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:29.906 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:29.900093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:39.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:39.978895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:40.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:40.027581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:50.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:50.147127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:32:50.164 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:32:50.158443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:00.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:00.292589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:00.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:00.307535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:10.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:10.438055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:10.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:10.450510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:20.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:20.541632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:20.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:20.555968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:30.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:30.671468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:30.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:30.684141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:40.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:40.795585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:40.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:40.807654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:50.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:50.909687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:33:50.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:33:50.923309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:01.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:01.039961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:01.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:01.053211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:11.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:11.173793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:11.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:11.188791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:21.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:21.290756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:21.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:21.304565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:31.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:31.372806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:31.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:31.386082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:41.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:41.489556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:41.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:41.503483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:51.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:51.594218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:34:51.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:34:51.606195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:01.696 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:01.689611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:01.763 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:01.757255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:11.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:11.849573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:11.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:11.864072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:21.981 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:21.974845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:21.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:21.988129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:32.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:32.112545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:32.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:32.125210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:42.230 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:42.223438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:42.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:42.255902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:52.334 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:52.327759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:35:52.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:35:52.340869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:02.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:02.406488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:02.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:02.420779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:12.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:12.487358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:12.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:12.504398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:22.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:22.616964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:22.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:22.630752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:32.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:32.707493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:32.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:32.719289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:42.791 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:42.780133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:42.808 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:42.800262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:50.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:50.973205 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T04:36:51.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:51.611499 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:36:51.600+0000 7fb6479be380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:36:51.619 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T04:36:51.600+0000 7fb6479be380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:36:51.619 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T04:36:51.600+0000 7fb6479be380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:36:51.619 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:36:52.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:52.903113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:36:52.921 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:36:52.915128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:02.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:02.984353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:03.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:02.998269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:13.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:13.095445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:13.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:13.112359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:23.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:23.187716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:23.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:23.201683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:33.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:33.270046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:33.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:33.283280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:43.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:43.348481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:43.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:43.416290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:53.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:53.491444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:37:53.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:37:53.509500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:03.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:03.622798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:03.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:03.635292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:13.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:13.699491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:13.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:13.711947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:23.818 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:23.811561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:23.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:23.825000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:33.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:33.932977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:33.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:33.947512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:44.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:44.012567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:44.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:44.070201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:54.202 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:54.195167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:38:54.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:38:54.207165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:04.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:04.332808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:04.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:04.346328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:14.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:14.481671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:14.502 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:14.495883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:24.572 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:24.565610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:24.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:24.579942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:34.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:34.670307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:34.691 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:34.685208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:44.796 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:44.790022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:44.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:44.806250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:54.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:54.872011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:39:54.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:39:54.886436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:04.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:04.978481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:04.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:04.993398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:15.094 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:15.088254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:15.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:15.104058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:25.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:25.215601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:25.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:25.229224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:35.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:35.334925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:35.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:35.390509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:45.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:45.478695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:45.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:45.491760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:55.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:55.622122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:40:55.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:40:55.649024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:05.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:05.715291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:05.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:05.772684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:15.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:15.856482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:15.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:15.871196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:25.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:25.942712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:25.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:25.956850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:36.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:36.067459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:36.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:36.081832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:46.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:46.214240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:46.233 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:46.226967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:56.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:56.298766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:41:56.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:41:56.312998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:06.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:06.423202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:06.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:06.439619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:16.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:16.527193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:16.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:16.539784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:26.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:26.643783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:26.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:26.657316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:36.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:36.768079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:36.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:36.783497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:46.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:46.902763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:46.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:46.919350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:57.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:56.998030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:42:57.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:42:57.010180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:07.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:07.084555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:07.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:07.099115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:17.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:17.207837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:17.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:17.221254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:27.334 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:27.328630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:27.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:27.342130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:37.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:37.408871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:37.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:37.423241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:47.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:47.524204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:47.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:47.538361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:57.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:57.599245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:43:57.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:43:57.614515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:07.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:07.728390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:07.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:07.741367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:17.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:17.841729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:17.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:17.854685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:27.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:27.919152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:27.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:27.933445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:38.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:38.047097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:38.065 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:38.059872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:48.141 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:48.136303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:48.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:48.175200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:58.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:58.270332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:44:58.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:44:58.282524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:08.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:08.362130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:08.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:08.377351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:18.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:18.495450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:18.513 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:18.508545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:28.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:28.578815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:28.596 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:28.591898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:38.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:38.738179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:38.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:38.754113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:48.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:48.831351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:48.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:48.881037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:58.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:58.948908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:45:58.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:45:58.963515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:09.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:09.031146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:09.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:09.046695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:19.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:19.180624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:19.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:19.194554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:29.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:29.343969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:29.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:29.363932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:39.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:39.452605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:39.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:39.464427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:49.566 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:49.561118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:49.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:49.575008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:59.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:59.648190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:46:59.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:46:59.660608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:09.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:09.727725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:09.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:09.741806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:19.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:19.810041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:19.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:19.823375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:29.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:29.899798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:29.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:29.912743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:39.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:39.993858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:40.014 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:40.009610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:50.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:50.072938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:47:50.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:47:50.092534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:00.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:00.167337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:00.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:00.180055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:10.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:10.289914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:10.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:10.302467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:20.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:20.410467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:20.428 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:20.423749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:30.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:30.546687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:30.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:30.560230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:40.657 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:40.652954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:40.669 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:40.665381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:50.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:50.729207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:48:50.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:48:50.778990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:00.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:00.852161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:00.912 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:00.908208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:11.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:11.052287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:11.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:11.065394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:21.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:21.178385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:21.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:21.194290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:31.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:31.286688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:31.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:31.300952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:41.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:41.371305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:41.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:41.385460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:51.502 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:51.497470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:49:51.514 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:49:51.510590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:01.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:01.612886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:01.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:01.628475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:11.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:11.701972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:11.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:11.753027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:21.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:21.824050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:21.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:21.873582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:32.014 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:32.009705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:32.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:32.023633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:42.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:42.137542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:42.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:42.150498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:52.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:52.215503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:50:52.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:50:52.269192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:02.358 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:02.353775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:02.370 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:02.366213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:12.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:12.469861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:12.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:12.482018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:22.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:22.576931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:22.594 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:22.590409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:32.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:32.696428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:32.715 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:32.711361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:42.786 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:42.781535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:42.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:42.794133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:52.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:52.873589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:51:52.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:51:52.887943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:03.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:03.044099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:03.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:03.057042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:13.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:13.162254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:13.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:13.176657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:23.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:23.259561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:23.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:23.272462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:33.377 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:33.372432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:33.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:33.386910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:43.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:43.450134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:43.463 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:43.459510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:53.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:53.533043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:52:53.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:52:53.545808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:03.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:03.645627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:03.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:03.658756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:13.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:13.746749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:13.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:13.760658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:23.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:23.863707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:23.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:23.877829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:31.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:31.622395 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T04:53:32.274 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:32.269883 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T04:53:32.258+0000 7fe417379380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T04:53:32.274 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T04:53:32.258+0000 7fe417379380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T04:53:32.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T04:53:32.258+0000 7fe417379380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T04:53:32.275 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T04:53:33.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:33.964877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:33.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:33.980638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:44.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:44.072963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:44.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:44.086825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:54.205 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:54.201679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:53:54.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:53:54.251144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:04.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:04.358617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:04.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:04.372365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:14.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:14.454982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:14.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:14.465959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:24.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:24.566117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:24.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:24.580495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:34.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:34.651703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:34.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:34.664247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:44.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:44.735711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:44.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:44.748463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:54.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:54.811652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:54:54.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:54:54.822483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:04.922 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:04.918787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:04.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:04.932986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:14.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:14.995141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:15.014 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:15.010800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:25.084 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:25.080382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:25.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:25.130596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:35.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:35.247046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:35.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:35.259561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:45.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:45.375348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:45.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:45.389842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:55.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:55.504512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:55:55.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:55:55.520975 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:05.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:05.617120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:05.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:05.629371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:15.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:15.705025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:15.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:15.720276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:25.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:25.824383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:25.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:25.835285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:35.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:35.919787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:35.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:35.934913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:46.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:46.082048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:46.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:46.092703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:56.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:56.163018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:56:56.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:56:56.175523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:06.260 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:06.256340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:06.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:06.269383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:16.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:16.339297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:16.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:16.353826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:26.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:26.431548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:26.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:26.442305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:36.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:36.519392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:36.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:36.534401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:46.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:46.633926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:46.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:46.648020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:56.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:56.732050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:57:56.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:57:56.746416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:06.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:06.816841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:06.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:06.875553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:16.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:16.984905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:17.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:16.997025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:27.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:27.077628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:27.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:27.089543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:37.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:37.194760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:37.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:37.205716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:47.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:47.293853 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:47.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:47.308255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:57.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:57.418249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:58:57.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:58:57.433093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:07.558 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:07.554096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:07.568 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:07.564711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:17.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:17.691607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:17.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:17.702931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:27.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:27.849657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:27.865 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:27.861728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:37.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:37.938251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:37.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:37.953686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:48.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:48.024418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:48.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:48.038102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:58.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:58.129429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T04:59:58.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 04:59:58.181447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:08.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:08.260348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:08.277 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:08.273723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:18.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:18.353125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:18.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:18.365802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:28.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:28.435025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:28.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:28.449348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:38.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:38.517604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:38.571 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:38.567547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:48.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:48.674182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:48.691 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:48.687937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:58.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:58.760445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:00:58.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:00:58.773532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:08.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:08.883788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:08.900 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:08.897000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:18.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:18.972706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:18.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:18.986947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:29.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:29.088538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:29.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:29.103338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:39.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:39.168911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:39.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:39.182268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:49.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:49.253408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:49.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:49.265085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:59.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:59.363655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:01:59.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:01:59.375685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:09.502 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:09.496913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:09.514 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:09.510921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:19.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:19.576023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:19.594 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:19.590904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:29.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:29.666812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:29.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:29.680840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:39.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:39.756710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:39.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:39.772809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:49.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:49.907265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:02:49.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:02:49.921209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:00.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:00.015184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:00.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:00.104104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:10.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:10.202964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:10.257 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:10.253574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:20.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:20.334823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:20.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:20.347079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:30.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:30.432597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:30.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:30.446963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:40.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:40.556381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:40.574 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:40.570762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:50.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:50.638374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:03:50.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:03:50.651433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:00.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:00.724003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:00.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:00.739142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:10.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:10.828085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:10.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:10.842293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:20.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:20.905696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:20.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:20.916844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:31.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:31.009437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:31.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:31.021587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:41.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:41.148378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:41.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:41.162664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:51.237 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:51.233725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:04:51.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:04:51.245030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:01.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:01.319851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:01.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:01.332854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:11.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:11.433665 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:11.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:11.445639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:21.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:21.522265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:21.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:21.536823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:31.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:31.655097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:31.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:31.668485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:41.806 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:41.803440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:41.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:41.812569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:51.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:51.888544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:05:51.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:05:51.900936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:02.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:02.006340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:02.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:02.024833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:12.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:12.088652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:12.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:12.184404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:22.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:22.314869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:22.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:22.326137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:32.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:32.451919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:32.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:32.466511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:42.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:42.593702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:42.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:42.606658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:52.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:52.723293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:06:52.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:06:52.737180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:02.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:02.811765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:02.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:02.822433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:12.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:12.934243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:12.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:12.950418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:23.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:23.024385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:23.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:23.037344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:33.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:33.102932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:33.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:33.124322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:43.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:43.232695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:43.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:43.244611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:53.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:53.336468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:07:53.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:07:53.351690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:03.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:03.421361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:03.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:03.435499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:13.527 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:13.523506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:13.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:13.538387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:23.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:23.654830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:23.669 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:23.666258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:33.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:33.763742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:33.781 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:33.777865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:43.850 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:43.847190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:43.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:43.900868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:53.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:53.995777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:08:54.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:08:54.008988 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:04.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:04.114587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:04.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:04.128309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:14.235 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:14.231983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:14.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:14.243921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:24.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:24.348813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:24.364 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:24.361645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:34.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:34.451435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:34.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:34.465653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:44.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:44.579749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:44.599 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:44.595807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:54.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:54.692418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:09:54.713 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:09:54.707492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:04.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:04.798969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:04.816 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:04.813264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:12.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:12.281136 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T05:10:12.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:12.928948 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:10:12.918+0000 7fc6a820b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:10:12.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T05:10:12.918+0000 7fc6a820b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:10:12.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T05:10:12.918+0000 7fc6a820b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:10:12.934 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:10:14.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:14.875257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:14.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:14.887746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:24.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:24.989314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:25.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:25.001545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:35.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:35.113714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:35.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:35.128451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:45.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:45.225054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:45.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:45.238495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:55.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:55.358935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:10:55.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:10:55.372362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:05.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:05.482555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:05.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:05.495893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:15.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:15.573563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:15.589 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:15.586436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:25.720 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:25.717493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:25.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:25.730924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:35.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:35.794407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:35.809 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:35.806908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:45.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:45.883510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:45.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:45.900966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:55.981 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:55.978647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:11:55.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:11:55.990806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:06.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:06.068993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:06.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:06.110070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:16.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:16.214009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:16.231 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:16.228865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:26.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:26.303732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:26.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:26.317419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:36.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:36.433767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:36.448 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:36.445917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:46.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:46.582437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:46.596 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:46.593946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:56.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:56.691897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:12:56.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:12:56.705784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:06.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:06.789880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:06.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:06.799582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:16.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:16.947481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:16.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:16.957463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:27.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:27.063969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:27.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:27.074564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:37.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:37.141841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:37.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:37.154263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:47.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:47.243209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:47.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:47.255895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:57.364 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:57.360883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:13:57.377 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:13:57.373976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:07.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:07.470653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:07.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:07.482536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:17.599 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:17.595616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:17.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:17.610607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:27.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:27.765345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:27.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:27.779967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:37.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:37.892316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:37.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:37.905003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:47.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:47.974431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:47.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:47.988462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:58.066 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:58.062229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:14:58.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:14:58.150707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:08.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:08.221990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:08.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:08.235387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:18.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:18.356503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:18.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:18.371057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:28.544 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:28.541535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:28.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:28.552099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:38.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:38.756145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:38.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:38.772976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:48.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:48.843831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:48.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:48.857224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:58.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:58.980063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:15:58.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:15:58.995309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:09.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:09.089541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:09.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:09.119641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:19.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:19.208685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:19.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:19.222658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:29.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:29.327090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:29.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:29.340199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:39.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:39.437161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:39.453 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:39.450024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:49.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:49.535968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:49.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:49.583398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:59.682 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:59.678974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:16:59.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:16:59.741274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:09.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:09.812124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:09.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:09.823534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:19.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:19.893352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:19.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:19.941218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:30.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:30.078079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:30.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:30.090802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:40.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:40.251012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:40.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:40.261401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:50.400 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:50.397131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:17:50.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:17:50.411380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:00.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:00.560088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:00.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:00.573207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:10.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:10.646634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:10.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:10.692583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:20.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:20.763109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:20.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:20.776558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:30.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:30.853663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:30.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:30.865529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:40.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:40.982441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:41.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:40.998139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:51.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:51.056029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:18:51.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:18:51.068411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:01.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:01.134902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:01.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:01.146857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:11.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:11.217626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:11.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:11.229690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:21.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:21.298091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:21.314 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:21.311434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:31.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:31.401567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:31.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:31.415383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:41.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:41.484179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:41.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:41.515299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:51.618 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:51.614810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:19:51.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:19:51.627028 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:01.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:01.734222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:01.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:01.749147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:11.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:11.823138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:11.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:11.836553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:21.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:21.912191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:21.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:21.927442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:32.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:32.026329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:32.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:32.041023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:42.113 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:42.110417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:42.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:42.150347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:52.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:52.256155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:20:52.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:20:52.272404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:02.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:02.388429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:02.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:02.403967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:12.485 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:12.482250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:12.500 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:12.497481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:22.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:22.619416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:22.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:22.632980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:32.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:32.731601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:32.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:32.746757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:42.850 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:42.847327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:42.861 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:42.858751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:52.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:52.987741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:21:53.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:21:52.997786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:03.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:03.057860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:03.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:03.069367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:13.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:13.143867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:13.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:13.162578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:23.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:23.251381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:23.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:23.264617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:33.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:33.408857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:33.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:33.421947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:43.527 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:43.524792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:43.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:43.537867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:53.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:53.652887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:22:53.669 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:22:53.666956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:03.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:03.735854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:03.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:03.750386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:13.831 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:13.828411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:13.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:13.841457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:23.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:23.915716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:23.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:23.928753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:33.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:33.996666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:34.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:34.009333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:44.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:44.087326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:44.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:44.129905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:54.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:54.222951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:23:54.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:23:54.237077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:04.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:04.346159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:04.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:04.360669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:14.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:14.435884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:14.453 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:14.450653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:24.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:24.548660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:24.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:24.568114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:34.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:34.641768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:34.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:34.653776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:44.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:44.757199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:44.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:44.769819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:54.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:54.873249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:24:54.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:24:54.887304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:04.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:04.965243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:05.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:05.016135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:15.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:15.123091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:15.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:15.137268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:25.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:25.218257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:25.233 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:25.230947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:35.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:35.344378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:35.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:35.359291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:45.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:45.458980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:45.475 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:45.472467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:55.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:55.546903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:25:55.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:25:55.562116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:05.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:05.661386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:05.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:05.675259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:15.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:15.774493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:15.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:15.786802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:25.861 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:25.858758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:25.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:25.873670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:36.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:36.002238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:36.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:36.014877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:46.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:46.095735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:46.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:46.182356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:52.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:52.937833 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T05:26:53.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:53.619273 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:26:53.605+0000 7f758379b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:26:53.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T05:26:53.605+0000 7f758379b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:26:53.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T05:26:53.605+0000 7f758379b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:26:53.623 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:26:56.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:56.250283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:26:56.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:26:56.261699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:06.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:06.350376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:06.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:06.365268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:16.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:16.438043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:16.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:16.451425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:26.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:26.528816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:26.603 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:26.600591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:36.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:36.721134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:36.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:36.733878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:46.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:46.874057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:46.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:46.886601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:56.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:56.990140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:27:57.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:27:57.010142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:07.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:07.114008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:07.130 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:07.128088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:17.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:17.216125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:17.235 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:17.232569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:27.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:27.385815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:27.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:27.399812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:37.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:37.518891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:37.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:37.532248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:47.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:47.628949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:47.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:47.638911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:57.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:57.752194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:28:57.770 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:28:57.767262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:07.865 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:07.862509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:07.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:07.875428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:17.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:17.943570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:18.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:18.006408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:28.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:28.103272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:28.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:28.116520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:38.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:38.247080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:38.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:38.259389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:48.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:48.357911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:48.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:48.373387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:58.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:58.493821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:29:58.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:29:58.509836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:08.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:08.582458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:08.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:08.598499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:18.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:18.707136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:18.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:18.721178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:28.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:28.875895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:28.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:28.888673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:39.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:39.021201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:39.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:39.036593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:49.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:49.146961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:49.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:49.159118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:59.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:59.228777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:30:59.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:30:59.240519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:09.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:09.360537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:09.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:09.372338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:19.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:19.520560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:19.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:19.535951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:29.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:29.640299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:29.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:29.653615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:39.770 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:39.767789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:39.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:39.783081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:49.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:49.855038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:49.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:49.868614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:59.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:59.977583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:31:59.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:31:59.992134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:10.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:10.111315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:10.130 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:10.125597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:20.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:20.205444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:20.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:20.217960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:30.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:30.286109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:30.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:30.299133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:40.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:40.440386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:40.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:40.452058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:50.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:50.536321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:32:50.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:32:50.550646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:00.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:00.641226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:00.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:00.657475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:10.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:10.782830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:10.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:10.806512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:20.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:20.878968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:20.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:20.893842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:31.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:30.999634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:31.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:31.010447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:41.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:41.126120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:41.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:41.136291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:51.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:51.284957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:33:51.298 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:33:51.296097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:01.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:01.364342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:01.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:01.380358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:11.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:11.446124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:11.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:11.459956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:21.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:21.528793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:21.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:21.595007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:31.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:31.837025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:31.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:31.853318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:41.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:41.931293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:42.008 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:42.006074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:52.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:52.072407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:34:52.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:34:52.086715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:02.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:02.208655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:02.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:02.221599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:12.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:12.323732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:12.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:12.336803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:22.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:22.411043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:22.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:22.423235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:32.513 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:32.510838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:32.526 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:32.523746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:42.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:42.631811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:42.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:42.645783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:52.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:52.763246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:35:52.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:35:52.777137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:02.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:02.877125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:02.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:02.889138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:12.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:12.986535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:13.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:12.998663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:23.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:23.094717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:23.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:23.109830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:33.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:33.193347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:33.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:33.206336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:43.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:43.310398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:43.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:43.327017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:53.446 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:53.443826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:36:53.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:36:53.457671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:03.544 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:03.541609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:03.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:03.556660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:13.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:13.639033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:13.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:13.671784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:23.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:23.786810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:23.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:23.798287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:33.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:33.986973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:34.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:34.000561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:44.074 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:44.071691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:44.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:44.087109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:54.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:54.156694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:37:54.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:37:54.169233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:04.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:04.275997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:04.293 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:04.290528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:14.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:14.442053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:14.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:14.456082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:24.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:24.525761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:24.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:24.579133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:34.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:34.653399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:34.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:34.665948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:44.773 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:44.769959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:44.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:44.781557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:54.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:54.834558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:38:54.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:38:54.848502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:04.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:04.925507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:04.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:04.941087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:15.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:15.040270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:15.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:15.052601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:25.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:25.130954 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:25.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:25.203667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:35.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:35.339627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:35.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:35.354344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:45.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:45.466235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:45.480 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:45.477507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:55.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:55.617213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:39:55.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:39:55.630565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:05.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:05.744312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:05.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:05.757972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:15.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:15.831839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:15.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:15.846305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:25.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:25.957387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:25.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:25.969146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:36.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:36.064219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:36.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:36.078720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:46.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:46.269887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:46.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:46.281954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:56.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:56.393979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:40:56.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:40:56.406153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:06.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:06.485722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:06.504 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:06.501262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:16.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:16.636590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:16.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:16.649545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:26.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:26.753869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:26.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:26.764949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:36.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:36.872903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:36.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:36.887149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:47.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:46.998067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:47.016 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:47.013514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:57.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:57.085880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:41:57.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:41:57.097440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:07.235 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:07.232618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:07.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:07.247788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:17.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:17.354690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:17.371 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:17.368122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:27.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:27.432893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:27.448 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:27.445510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:37.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:37.577838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:37.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:37.592860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:47.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:47.659528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:47.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:47.673259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:57.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:57.734389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:42:57.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:42:57.749052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:07.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:07.843306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:07.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:07.856836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:17.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:17.928191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:17.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:17.974485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:28.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:28.045672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:28.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:28.059151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:33.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:33.629150 I | op-mon: parsing mon endpoints: a=10.99.228.129:6789,b=10.101.240.174:6789,c=10.111.0.148:6789 2022-03-10T05:43:34.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:34.283486 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-10T05:43:34.269+0000 7f1a8fa00380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-10T05:43:34.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T05:43:34.269+0000 7f1a8fa00380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-10T05:43:34.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10T05:43:34.269+0000 7f1a8fa00380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-10T05:43:34.287 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-10T05:43:38.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:38.169940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:38.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:38.182798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:48.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:48.254924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:48.269 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:48.266195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:58.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:58.376117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:43:58.394 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:43:58.391703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:08.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:08.493943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:08.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:08.505915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:18.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:18.579709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:18.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:18.592718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:28.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:28.706738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:28.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:28.721997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:38.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:38.814947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:38.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:38.899519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:48.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:48.980823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:48.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:48.995006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:59.074 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:59.071487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:44:59.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:44:59.083806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:09.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:09.154110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:09.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:09.168019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:19.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:19.236598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:19.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:19.248649 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:29.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:29.338443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:29.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:29.350394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:39.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:39.458614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:39.475 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:39.473179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:49.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:49.627970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:49.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:49.641180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:59.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:59.735056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:45:59.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:45:59.748916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:09.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:09.834957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:09.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:09.849881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:19.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:19.931468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:19.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:19.994782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:30.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:30.149348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:30.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:30.167607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:40.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:40.285501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:40.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:40.296691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:50.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:50.372306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:46:50.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:46:50.386314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:00.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:00.469377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:00.484 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:00.481433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:10.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:10.590262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:10.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:10.603324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:20.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:20.682600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:20.757 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:20.755245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:30.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:30.856037 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:30.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:30.867860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:40.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:40.974388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:40.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:40.989282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:51.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:51.088513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:47:51.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:47:51.103213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:01.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:01.261002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:01.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:01.271188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:11.358 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:11.355940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:11.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:11.369974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:21.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:21.516943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:21.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:21.527842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:31.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:31.601942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:31.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:31.615047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:41.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:41.718722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:41.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:41.732970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:51.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:51.851066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:48:51.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:48:51.866190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:01.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:01.985073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:01.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:01.996856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:12.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:12.094918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:12.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:12.107274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:22.181 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:22.178821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:22.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:22.191450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:32.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:32.314248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:32.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:32.331486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:42.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:42.394215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:42.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:42.406626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:52.511 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:52.508506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:49:52.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:49:52.523272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:02.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:02.584146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:02.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:02.598231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:12.697 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:12.695057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:12.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:12.709030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:22.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:22.787522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:22.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:22.799025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:32.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:32.906610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:32.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:32.921513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:43.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:43.022455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:43.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:43.035871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:53.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:53.134992 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:50:53.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:50:53.148732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:03.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:03.253384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:03.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:03.265205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:13.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:13.375113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:13.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:13.392396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:23.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:23.463558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:23.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:23.475705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:33.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:33.600586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:33.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:33.613491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:43.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:43.724008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:43.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:43.738695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:53.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:53.817803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:51:53.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:51:53.841851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:03.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:52:03.969616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:03.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:52:03.983734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:14.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:52:14.121919 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T05:52:14.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-10 05:52:14.133706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-10T08:13:31.001 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-10T08:13:31.020 DEBUG:teuthology.task.console_log:Killing console logger for smithi007 2022-03-10T08:13:31.021 DEBUG:teuthology.task.console_log:Killing console logger for smithi071 2022-03-10T08:13:31.021 DEBUG:teuthology.task.console_log:Killing console logger for smithi184 2022-03-10T08:13:31.022 DEBUG:teuthology.exit:Finished running handlers