2022-03-06T00:11:26.140 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-06T00:11:26.142 DEBUG:teuthology.run:Teuthology command: teuthology --name ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi --owner scheduled_ozeneva@teuthology --verbose --archive /home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723221 --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/host rook/master} -- /home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723221/orig.config.yaml 2022-03-06T00:11:26.206 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-06T00:11:26.269 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723221 branch: wip-omri-tracing-compiled description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/host rook/master} email: null first_in_suite: false job_id: '6723221' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-omri-tracing-compiled ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 spec: mon: allowMultiplePerNode: false network: provider: host ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 kernel: hwe: true kubeadm: version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-omri-tracing-compiled sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 owner: scheduled_ozeneva@teuthology priority: 50 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 sleep_before_teardown: 0 suite: rados suite_branch: wip-omri-tracing-compiled suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 targets: smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUvVfVAcBY7fJK4FUig29uXopb6NQ8keCVxtkL1E+56RZ7kLgiq/6y+Ueh46fFVGIem4CzRxUwvmiQf8ia9V6yKif7bCq4+CqEiNleBlaSd99vkxTxqAh+UW9/duaR1FAgdfmn8xFjXXvwFYpXMrZxCH+tiEJMWYyzs7c209auOZqK4apCUNYPbh8jERLAyFLrDTaeJhCT0b9yzot1LkZreVBJ1zxWqaZ51Dwn/uwZPrHFV41hYWnOT8BjzbswB8/5tSNQWNtGMKtP0vye09np+ivlDW5bJzsegXWO+wGFIdzwnRhva7ZHNCP8IrKtutaTNzoZGDCBjrms+7E/PvUrqtSU6ouH9qj0tE3nlIUnAX65nVFeivQGDkjd9mxuNb9n87/fVbsyctTAqQE3SC+A5PbwHq/+O+tduakZKpbWXyWFfnfhI2F0CaWPMKgrd3B7eVxGgn3xDw8Bbm7JvALJbkemZCdQWOJOwv3eomZ2xDzma+rsj/X9X+ZJmYWOe6M= smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtyGs/NtePQLqqNR0HVkHyp0Gk6yce0f1xnIxRtXvtsKHeZjv6zv6nTGUIsD7EXsiT2XKF8ySqEeeq05jPqJvUBDuNj8N3xfHn/97qkGqO8LXEzYgawZj8xAo7TeYeLjEV+mCJOK8qqeEgXjT0pTIvm9SDcq/JNE9OnYL/2Vy+fHHgw5zymBmHcNt8yTPfGwJzAnCmpVTjFKwZljLbYxOvBygKhFKs6Een7GhI1XHyHIVT/v0Cghi+ewzdM6Bc9sIcbMpWah9NFiHzreWOeuY/goq0NcSopfdijF+qDhrIiPT0u3/emf5C0xlNB1qjvsK3sVL4js/QKxAQMUEiISTRJ8b09cugW/ci3xKwl9OFqeNVjrcQijQ0Ne3+VuRaR/9nHkdD825C6xJl3PV6oTLXFLidgLL0Q2+rUcgyHUGeinIGcYADSrzU4kPSZXZPfDVOZEtzhc5uE+PjGGXm1WfAo+pkp+u0rseDtxO2GER0CZBDEJ/ZWvFViQTC8by6cbc= smithi059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5bH4Pr5YSrOpFCISKDPeNV9/VQaT4SvFGBG588WfMLss5lGvAA+wj9AT8zgH+4r+j9bqSsvnIu0hpOPRX6QkFiF1nyT8ll8EkaQ7KsE23akDrdE5Gc7YT+8yVFI2v+utyaoeScXczpD+8xvGkDWnsYZOVu/eYNRto9FSyyGM+hHY9rmu/U+XqE2usrgBj6HjetlgNBJvh8t4ck7c/XNK8yIoWkR9aa0QnLjvRQ1lanUosXRlJ/lKhQYf75nZEDeUQ8qsIDP5Wsc7LbLg4mGvPMdHdFcQlNrvfDCLmseyKYOTMo824UA5mi0x2bQgQilFqyaOQdldVwXZ7jTox3tOknRI7xoY/EMeGQUkStPaGQB0KXIAtjy7VT3JoIiutzWXKqNBLtUJYs+S9+z9OWz0TiEc+P//OR9xiQg/tgxnq4IHhxTC2Xjp2bOOMGWbmMlVslN+7Z76nC2x71L26NTRuh4KkOloR2qMaj+q2aK5ZfoaqoktndRIMtOh40qQQxyM= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-06_00:00:14 tube: smithi user: ozeneva verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-06T00:11:26.269 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa; will attempt to use it 2022-03-06T00:11:26.270 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks 2022-03-06T00:11:26.270 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-06T00:11:26.272 INFO:teuthology.task.internal:Checking packages... 2022-03-06T00:11:26.294 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd0dc2fbbf3f64a2d513e2355b245d0ea9728ff97' 2022-03-06T00:11:26.294 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-06T00:11:26.295 INFO:teuthology.packaging:ref: None 2022-03-06T00:11:26.295 INFO:teuthology.packaging:tag: None 2022-03-06T00:11:26.295 INFO:teuthology.packaging:branch: wip-omri-tracing-compiled 2022-03-06T00:11:26.295 INFO:teuthology.packaging:sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 2022-03-06T00:11:26.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-omri-tracing-compiled 2022-03-06T00:11:26.475 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-10974-gd0dc2fbb-1focal 2022-03-06T00:11:26.476 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-06T00:11:26.521 INFO:teuthology.task.internal:no buildpackages task found 2022-03-06T00:11:26.522 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-06T00:11:26.864 INFO:teuthology.task.internal:Saving configuration 2022-03-06T00:11:26.876 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-06T00:11:27.134 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-06T00:11:27.157 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723221', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-06 00:06:26.303117', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII7/11MGG7SWAREnSOCKqlA4otaKbe+jHB2HVM/w2ZHA'} 2022-03-06T00:11:27.179 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723221', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-06 00:06:26.301634', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKAEaKh+S2dwwUE6/csH+o9K3zAZ7jgkaIutDwCFS9Jx'} 2022-03-06T00:11:27.202 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723221', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-06 00:06:26.304183', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMs3jtKGDaOlppyS37p08Nop2V4vc/DSKj0tVvU8n3iC'} 2022-03-06T00:11:27.202 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-06T00:11:27.243 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-06T00:11:27.243 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-06T00:11:27.243 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-06T00:11:27.244 INFO:teuthology.run_tasks:Running task console_log... 2022-03-06T00:11:27.460 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2972a4c950>, signals=[15]) 2022-03-06T00:11:27.462 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-06T00:11:27.503 INFO:teuthology.task.internal:Opening connections... 2022-03-06T00:11:27.503 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com 2022-03-06T00:11:27.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:11:27.567 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2022-03-06T00:11:27.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:11:27.620 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2022-03-06T00:11:27.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:11:27.677 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-06T00:11:27.691 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2022-03-06T00:11:27.700 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2022-03-06T00:11:27.700 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2022-03-06T00:11:27.748 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="Ubuntu" 2022-03-06T00:11:27.748 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-06T00:11:27.749 INFO:teuthology.orchestra.run.smithi018.stdout:ID=ubuntu 2022-03-06T00:11:27.749 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE=debian 2022-03-06T00:11:27.749 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-06T00:11:27.750 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="20.04" 2022-03-06T00:11:27.750 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-06T00:11:27.750 INFO:teuthology.orchestra.run.smithi018.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-06T00:11:27.751 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-06T00:11:27.751 INFO:teuthology.orchestra.run.smithi018.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-06T00:11:27.751 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_CODENAME=focal 2022-03-06T00:11:27.751 INFO:teuthology.orchestra.run.smithi018.stdout:UBUNTU_CODENAME=focal 2022-03-06T00:11:27.752 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2022-03-06T00:11:27.773 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2022-03-06T00:11:27.780 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2022-03-06T00:11:27.780 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2022-03-06T00:11:27.827 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="Ubuntu" 2022-03-06T00:11:27.827 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-06T00:11:27.828 INFO:teuthology.orchestra.run.smithi019.stdout:ID=ubuntu 2022-03-06T00:11:27.828 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE=debian 2022-03-06T00:11:27.828 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-06T00:11:27.829 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="20.04" 2022-03-06T00:11:27.829 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-06T00:11:27.829 INFO:teuthology.orchestra.run.smithi019.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-06T00:11:27.831 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-06T00:11:27.831 INFO:teuthology.orchestra.run.smithi019.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-06T00:11:27.831 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_CODENAME=focal 2022-03-06T00:11:27.831 INFO:teuthology.orchestra.run.smithi019.stdout:UBUNTU_CODENAME=focal 2022-03-06T00:11:27.832 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2022-03-06T00:11:27.850 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2022-03-06T00:11:27.858 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2022-03-06T00:11:27.859 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2022-03-06T00:11:27.907 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="Ubuntu" 2022-03-06T00:11:27.908 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-06T00:11:27.908 INFO:teuthology.orchestra.run.smithi059.stdout:ID=ubuntu 2022-03-06T00:11:27.908 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE=debian 2022-03-06T00:11:27.909 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-06T00:11:27.909 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="20.04" 2022-03-06T00:11:27.909 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-06T00:11:27.910 INFO:teuthology.orchestra.run.smithi059.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-06T00:11:27.910 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-06T00:11:27.910 INFO:teuthology.orchestra.run.smithi059.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-06T00:11:27.910 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_CODENAME=focal 2022-03-06T00:11:27.911 INFO:teuthology.orchestra.run.smithi059.stdout:UBUNTU_CODENAME=focal 2022-03-06T00:11:27.911 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2022-03-06T00:11:27.929 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-06T00:11:27.996 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-06T00:11:28.142 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-06T00:11:28.143 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest 2022-03-06T00:11:28.146 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2022-03-06T00:11:28.150 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2022-03-06T00:11:28.155 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-06T00:11:28.258 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-06T00:11:28.259 DEBUG:teuthology.orchestra.run.smithi018:> test -z $(ls -A /var/lib/ceph) 2022-03-06T00:11:28.261 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2022-03-06T00:11:28.264 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2022-03-06T00:11:28.281 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-06T00:11:28.488 INFO:teuthology.run_tasks:Running task kernel... 2022-03-06T00:11:28.609 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-06T00:11:28.610 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-06T00:11:28.611 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-06T00:11:28.611 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-06T00:11:28.612 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2022-03-06T00:11:28.614 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2022-03-06T00:11:28.615 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2022-03-06T00:11:28.623 INFO:teuthology.orchestra.run.smithi019.stdout:5.4.0-97-generic 2022-03-06T00:11:28.624 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.4.0-97-generic 2022-03-06T00:11:28.624 DEBUG:teuthology.orchestra.run.smithi019:> sudo apt-get clean 2022-03-06T00:11:28.628 INFO:teuthology.orchestra.run.smithi059.stdout:5.4.0-97-generic 2022-03-06T00:11:28.628 INFO:teuthology.orchestra.run.smithi018.stdout:5.4.0-97-generic 2022-03-06T00:11:28.629 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.4.0-97-generic 2022-03-06T00:11:28.630 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2022-03-06T00:11:28.630 INFO:teuthology.task.kernel:Running kernel on smithi018: 5.4.0-97-generic 2022-03-06T00:11:28.631 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get clean 2022-03-06T00:11:28.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2022-03-06T00:11:28.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo apt-get update 2022-03-06T00:11:28.744 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2022-03-06T00:11:28.884 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:11:28.894 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:11:28.994 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:11:29.000 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:11:29.019 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:11:29.019 INFO:teuthology.orchestra.run.smithi019.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:11:29.076 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:11:29.082 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:11:29.085 INFO:teuthology.orchestra.run.smithi019.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:11:29.162 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:11:29.198 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-06T00:11:29.284 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-06T00:11:29.287 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:11:29.291 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-06T00:11:29.296 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,740 B] 2022-03-06T00:11:29.297 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-06T00:11:29.304 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-06T00:11:29.304 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-06T00:11:29.305 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-06T00:11:29.306 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:11:29.313 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-06T00:11:29.319 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-06T00:11:29.321 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-06T00:11:29.327 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-06T00:11:29.330 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,740 B] 2022-03-06T00:11:29.330 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-06T00:11:29.331 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-06T00:11:29.335 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-06T00:11:29.336 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-06T00:11:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-06T00:11:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-06T00:11:29.341 INFO:teuthology.orchestra.run.smithi018.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-06T00:11:29.342 INFO:teuthology.orchestra.run.smithi018.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-06T00:11:29.343 INFO:teuthology.orchestra.run.smithi018.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-06T00:11:29.349 INFO:teuthology.orchestra.run.smithi018.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:11:29.355 INFO:teuthology.orchestra.run.smithi018.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-06T00:11:29.356 INFO:teuthology.orchestra.run.smithi018.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-06T00:11:29.368 INFO:teuthology.orchestra.run.smithi018.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-06T00:11:29.369 INFO:teuthology.orchestra.run.smithi018.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-06T00:11:29.369 INFO:teuthology.orchestra.run.smithi018.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-06T00:11:29.370 INFO:teuthology.orchestra.run.smithi018.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-06T00:11:29.420 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:11:29.428 INFO:teuthology.orchestra.run.smithi019.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:11:29.433 INFO:teuthology.orchestra.run.smithi018.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:11:29.496 INFO:teuthology.orchestra.run.smithi019.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:11:29.553 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-06T00:11:29.578 INFO:teuthology.orchestra.run.smithi018.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-06T00:11:29.639 INFO:teuthology.orchestra.run.smithi019.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-06T00:11:29.809 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-06T00:11:29.889 INFO:teuthology.orchestra.run.smithi018.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-06T00:11:29.907 INFO:teuthology.orchestra.run.smithi019.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-06T00:11:29.907 INFO:teuthology.orchestra.run.smithi019.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-06T00:11:29.922 INFO:teuthology.orchestra.run.smithi019.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-06T00:11:29.927 INFO:teuthology.orchestra.run.smithi018.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-06T00:11:29.932 INFO:teuthology.orchestra.run.smithi019.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,740 B] 2022-03-06T00:11:29.932 INFO:teuthology.orchestra.run.smithi019.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-06T00:11:29.942 INFO:teuthology.orchestra.run.smithi018.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-06T00:11:29.943 INFO:teuthology.orchestra.run.smithi018.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-06T00:11:29.944 INFO:teuthology.orchestra.run.smithi018.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-06T00:11:29.953 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-06T00:11:29.965 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-06T00:11:29.965 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-06T00:11:29.966 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-06T00:11:29.976 INFO:teuthology.orchestra.run.smithi018.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-06T00:11:29.979 INFO:teuthology.orchestra.run.smithi018.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-06T00:11:29.980 INFO:teuthology.orchestra.run.smithi018.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB] 2022-03-06T00:11:29.990 INFO:teuthology.orchestra.run.smithi019.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-06T00:11:29.990 INFO:teuthology.orchestra.run.smithi019.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-06T00:11:29.995 INFO:teuthology.orchestra.run.smithi019.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-06T00:11:29.995 INFO:teuthology.orchestra.run.smithi019.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:11:30.002 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-06T00:11:30.006 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-06T00:11:30.007 INFO:teuthology.orchestra.run.smithi018.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [669 kB] 2022-03-06T00:11:30.007 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB] 2022-03-06T00:11:30.023 INFO:teuthology.orchestra.run.smithi018.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-06T00:11:30.024 INFO:teuthology.orchestra.run.smithi019.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-06T00:11:30.035 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [669 kB] 2022-03-06T00:11:30.044 INFO:teuthology.orchestra.run.smithi019.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-06T00:11:30.048 INFO:teuthology.orchestra.run.smithi019.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-06T00:11:30.049 INFO:teuthology.orchestra.run.smithi019.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-06T00:11:30.056 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-06T00:11:30.056 INFO:teuthology.orchestra.run.smithi018.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-06T00:11:30.057 INFO:teuthology.orchestra.run.smithi018.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-06T00:11:30.057 INFO:teuthology.orchestra.run.smithi018.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-06T00:11:30.057 INFO:teuthology.orchestra.run.smithi018.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-06T00:11:30.059 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-06T00:11:30.061 INFO:teuthology.orchestra.run.smithi019.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-06T00:11:30.062 INFO:teuthology.orchestra.run.smithi019.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-06T00:11:30.089 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-06T00:11:30.090 INFO:teuthology.orchestra.run.smithi019.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-06T00:11:30.090 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-06T00:11:30.091 INFO:teuthology.orchestra.run.smithi019.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-06T00:11:30.091 INFO:teuthology.orchestra.run.smithi019.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-06T00:11:30.092 INFO:teuthology.orchestra.run.smithi019.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-06T00:11:30.092 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-06T00:11:30.092 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-06T00:11:30.092 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-06T00:11:30.098 INFO:teuthology.orchestra.run.smithi019.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-06T00:11:30.099 INFO:teuthology.orchestra.run.smithi019.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-06T00:11:30.103 INFO:teuthology.orchestra.run.smithi019.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-06T00:11:30.103 INFO:teuthology.orchestra.run.smithi019.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB] 2022-03-06T00:11:30.110 INFO:teuthology.orchestra.run.smithi018.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-06T00:11:30.111 INFO:teuthology.orchestra.run.smithi018.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-06T00:11:30.111 INFO:teuthology.orchestra.run.smithi018.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-06T00:11:30.112 INFO:teuthology.orchestra.run.smithi018.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-06T00:11:30.112 INFO:teuthology.orchestra.run.smithi018.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-06T00:11:30.118 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-06T00:11:30.119 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-06T00:11:30.132 INFO:teuthology.orchestra.run.smithi019.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [669 kB] 2022-03-06T00:11:30.139 INFO:teuthology.orchestra.run.smithi018.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-06T00:11:30.152 INFO:teuthology.orchestra.run.smithi019.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-06T00:11:30.155 INFO:teuthology.orchestra.run.smithi019.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-06T00:11:30.187 INFO:teuthology.orchestra.run.smithi019.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-06T00:11:30.187 INFO:teuthology.orchestra.run.smithi019.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-06T00:11:30.188 INFO:teuthology.orchestra.run.smithi019.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-06T00:11:30.188 INFO:teuthology.orchestra.run.smithi019.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-06T00:11:30.188 INFO:teuthology.orchestra.run.smithi019.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-06T00:11:30.202 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-06T00:11:30.202 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-06T00:11:30.203 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-06T00:11:30.203 INFO:teuthology.orchestra.run.smithi059.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-06T00:11:30.214 INFO:teuthology.orchestra.run.smithi019.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-06T00:11:30.220 INFO:teuthology.orchestra.run.smithi018.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-06T00:11:30.220 INFO:teuthology.orchestra.run.smithi018.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-06T00:11:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-06T00:11:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-06T00:11:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-06T00:11:30.313 INFO:teuthology.orchestra.run.smithi019.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-06T00:11:30.313 INFO:teuthology.orchestra.run.smithi019.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-06T00:11:32.587 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 10.0 MB in 2s (5,778 kB/s) 2022-03-06T00:11:32.695 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 10.0 MB in 2s (5,861 kB/s) 2022-03-06T00:11:33.022 INFO:teuthology.orchestra.run.smithi019.stdout:Fetched 10.0 MB in 2s (4,633 kB/s) 2022-03-06T00:11:33.758 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-03-06T00:11:33.782 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:11:33.856 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-03-06T00:11:34.033 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-03-06T00:11:34.033 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-03-06T00:11:34.059 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-06T00:11:34.083 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:11:34.122 INFO:teuthology.orchestra.run.smithi019.stdout:Reading package lists... 2022-03-06T00:11:34.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:11:34.155 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-06T00:11:34.189 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2022-03-06T00:11:34.189 INFO:teuthology.orchestra.run.smithi059.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:11:34.189 INFO:teuthology.orchestra.run.smithi059.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-06T00:11:34.190 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-06T00:11:34.190 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2022-03-06T00:11:34.191 INFO:teuthology.orchestra.run.smithi059.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-06T00:11:34.191 INFO:teuthology.orchestra.run.smithi059.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-06T00:11:34.215 INFO:teuthology.orchestra.run.smithi019.stdout:Reading package lists... 2022-03-06T00:11:34.238 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2022-03-06T00:11:34.239 INFO:teuthology.orchestra.run.smithi059.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:11:34.239 INFO:teuthology.orchestra.run.smithi059.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-06T00:11:34.239 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-06T00:11:34.240 INFO:teuthology.orchestra.run.smithi059.stdout: thermald upower usbmuxd 2022-03-06T00:11:34.331 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-06T00:11:34.332 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-06T00:11:34.394 INFO:teuthology.orchestra.run.smithi019.stdout:Building dependency tree... 2022-03-06T00:11:34.395 INFO:teuthology.orchestra.run.smithi019.stdout:Reading state information... 2022-03-06T00:11:34.421 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 13 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:11:34.422 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 80.1 MB of archives. 2022-03-06T00:11:34.422 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-06T00:11:34.423 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-06T00:11:34.480 INFO:teuthology.orchestra.run.smithi018.stdout:The following additional packages will be installed: 2022-03-06T00:11:34.481 INFO:teuthology.orchestra.run.smithi018.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:11:34.481 INFO:teuthology.orchestra.run.smithi018.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-06T00:11:34.482 INFO:teuthology.orchestra.run.smithi018.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-06T00:11:34.482 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2022-03-06T00:11:34.483 INFO:teuthology.orchestra.run.smithi018.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-06T00:11:34.483 INFO:teuthology.orchestra.run.smithi018.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-06T00:11:34.535 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2022-03-06T00:11:34.537 INFO:teuthology.orchestra.run.smithi018.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:11:34.537 INFO:teuthology.orchestra.run.smithi018.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-06T00:11:34.537 INFO:teuthology.orchestra.run.smithi018.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-06T00:11:34.538 INFO:teuthology.orchestra.run.smithi018.stdout: thermald upower usbmuxd 2022-03-06T00:11:34.544 INFO:teuthology.orchestra.run.smithi019.stdout:The following additional packages will be installed: 2022-03-06T00:11:34.545 INFO:teuthology.orchestra.run.smithi019.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:11:34.545 INFO:teuthology.orchestra.run.smithi019.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-06T00:11:34.545 INFO:teuthology.orchestra.run.smithi019.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-06T00:11:34.546 INFO:teuthology.orchestra.run.smithi019.stdout:Suggested packages: 2022-03-06T00:11:34.547 INFO:teuthology.orchestra.run.smithi019.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-06T00:11:34.547 INFO:teuthology.orchestra.run.smithi019.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-06T00:11:34.598 INFO:teuthology.orchestra.run.smithi019.stdout:The following NEW packages will be installed: 2022-03-06T00:11:34.600 INFO:teuthology.orchestra.run.smithi019.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:11:34.601 INFO:teuthology.orchestra.run.smithi019.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-06T00:11:34.602 INFO:teuthology.orchestra.run.smithi019.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-06T00:11:34.603 INFO:teuthology.orchestra.run.smithi019.stdout: thermald upower usbmuxd 2022-03-06T00:11:34.687 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-06T00:11:34.700 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-06T00:11:34.709 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-06T00:11:34.720 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 13 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:11:34.720 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 80.1 MB of archives. 2022-03-06T00:11:34.720 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-06T00:11:34.720 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-06T00:11:34.782 INFO:teuthology.orchestra.run.smithi019.stdout:0 upgraded, 13 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:11:34.783 INFO:teuthology.orchestra.run.smithi019.stdout:Need to get 80.1 MB of archives. 2022-03-06T00:11:34.783 INFO:teuthology.orchestra.run.smithi019.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-06T00:11:34.783 INFO:teuthology.orchestra.run.smithi019.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-06T00:11:34.791 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-06T00:11:34.792 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-06T00:11:34.990 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-06T00:11:35.002 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-06T00:11:35.003 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-06T00:11:35.045 INFO:teuthology.orchestra.run.smithi019.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-06T00:11:35.057 INFO:teuthology.orchestra.run.smithi019.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-06T00:11:35.058 INFO:teuthology.orchestra.run.smithi019.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-06T00:11:35.091 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-06T00:11:35.100 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-06T00:11:35.158 INFO:teuthology.orchestra.run.smithi019.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-06T00:11:35.164 INFO:teuthology.orchestra.run.smithi019.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-06T00:11:35.718 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-06T00:11:36.059 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-06T00:11:36.090 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-06T00:11:36.190 INFO:teuthology.orchestra.run.smithi019.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-06T00:11:36.450 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-06T00:11:36.545 INFO:teuthology.orchestra.run.smithi019.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-06T00:11:38.280 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-06T00:11:39.261 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-06T00:11:39.264 INFO:teuthology.orchestra.run.smithi019.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-06T00:11:39.264 INFO:teuthology.orchestra.run.smithi019.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-06T00:11:39.265 INFO:teuthology.orchestra.run.smithi019.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-06T00:11:39.265 INFO:teuthology.orchestra.run.smithi019.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-06T00:11:39.265 INFO:teuthology.orchestra.run.smithi019.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-06T00:11:39.265 INFO:teuthology.orchestra.run.smithi019.stdout:Fetched 80.1 MB in 4s (19.2 MB/s) 2022-03-06T00:11:39.266 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-06T00:11:39.266 INFO:teuthology.orchestra.run.smithi018.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-06T00:11:39.266 INFO:teuthology.orchestra.run.smithi018.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-06T00:11:39.266 INFO:teuthology.orchestra.run.smithi018.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-06T00:11:39.266 INFO:teuthology.orchestra.run.smithi018.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-06T00:11:39.267 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 80.1 MB in 4s (19.6 MB/s) 2022-03-06T00:11:39.267 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-06T00:11:39.268 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-06T00:11:39.268 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-06T00:11:39.268 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 80.1 MB in 4s (19.8 MB/s) 2022-03-06T00:11:39.268 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-06T00:11:39.269 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-06T00:11:39.364 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-06T00:11:40.368 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-06T00:11:40.372 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-06T00:11:40.496 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:11:40.789 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-06T00:11:40.792 INFO:teuthology.orchestra.run.smithi019.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-06T00:11:40.795 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-06T00:11:40.798 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-06T00:11:40.899 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-06T00:11:40.913 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-06T00:11:40.919 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:11:40.945 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:11:40.957 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:11:41.243 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-06T00:11:41.256 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-06T00:11:41.272 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-06T00:11:41.285 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-06T00:11:41.298 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-06T00:11:41.312 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-06T00:11:41.314 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:11:41.356 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:11:41.402 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:11:41.625 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-06T00:11:41.640 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-06T00:11:41.675 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-06T00:11:41.683 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:11:41.688 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-06T00:11:41.717 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:11:41.754 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-06T00:11:41.767 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-06T00:11:41.805 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:11:42.062 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-06T00:11:42.075 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-06T00:11:42.137 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:11:42.145 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-06T00:11:42.149 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-06T00:11:42.157 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-06T00:11:42.162 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-06T00:11:42.194 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:11:42.215 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:11:42.506 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-06T00:11:42.518 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-06T00:11:42.519 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-06T00:11:42.530 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-06T00:11:42.531 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:42.543 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-06T00:11:42.556 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:11:42.568 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:42.580 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:11:42.875 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-06T00:11:42.883 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-06T00:11:42.891 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:42.897 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:42.925 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:42.941 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:44.743 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-06T00:11:44.756 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:44.855 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:45.209 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-06T00:11:45.222 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:45.267 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-06T00:11:45.280 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:45.288 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-06T00:11:45.301 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:45.314 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:45.329 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:45.381 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:45.804 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-06T00:11:45.817 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:45.862 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:45.895 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-06T00:11:45.908 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:11:45.945 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:51.496 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-06T00:11:51.512 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-06T00:11:51.514 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-06T00:11:51.527 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-06T00:11:51.554 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:11:51.561 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:11:51.899 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-06T00:11:51.915 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-06T00:11:51.956 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:11:52.206 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-06T00:11:52.221 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-06T00:11:52.229 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-06T00:11:52.244 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-06T00:11:52.281 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:11:52.304 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:11:52.350 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package thermald. 2022-03-06T00:11:52.367 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-06T00:11:52.408 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:11:52.631 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package thermald. 2022-03-06T00:11:52.646 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-06T00:11:52.648 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-06T00:11:52.665 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-06T00:11:52.689 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:11:52.706 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:11:52.965 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package upower. 2022-03-06T00:11:52.979 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-06T00:11:53.033 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-06T00:11:53.066 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package thermald. 2022-03-06T00:11:53.081 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-06T00:11:53.116 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:11:53.202 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package upower. 2022-03-06T00:11:53.223 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-06T00:11:53.265 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-06T00:11:53.439 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package usbmuxd. 2022-03-06T00:11:53.452 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-06T00:11:53.489 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:11:53.657 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package upower. 2022-03-06T00:11:53.671 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-06T00:11:53.684 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package usbmuxd. 2022-03-06T00:11:53.698 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-06T00:11:53.743 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:11:53.743 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-06T00:11:53.804 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:11:53.954 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:11:54.043 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:11:54.071 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:11:54.155 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package usbmuxd. 2022-03-06T00:11:54.167 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:11:54.172 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-06T00:11:54.199 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:11:54.214 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:11:54.268 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:11:54.324 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:11:54.386 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:11:54.466 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:11:54.512 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:11:54.523 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:11:54.592 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-06T00:11:54.620 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:11:54.647 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:11:54.771 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:11:54.789 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-06T00:11:54.897 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:11:55.028 INFO:teuthology.orchestra.run.smithi059.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-06T00:11:55.050 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:11:55.077 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:11:55.199 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:11:55.248 INFO:teuthology.orchestra.run.smithi059.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-06T00:11:55.252 INFO:teuthology.orchestra.run.smithi019.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-06T00:11:55.258 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-06T00:11:55.258 INFO:teuthology.orchestra.run.smithi059.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-06T00:11:55.307 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:11:55.325 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-06T00:11:55.488 INFO:teuthology.orchestra.run.smithi019.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-06T00:11:55.494 INFO:teuthology.orchestra.run.smithi019.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-06T00:11:55.495 INFO:teuthology.orchestra.run.smithi019.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-06T00:11:55.769 INFO:teuthology.orchestra.run.smithi059.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-06T00:11:55.785 INFO:teuthology.orchestra.run.smithi018.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-06T00:11:55.843 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:11:55.990 INFO:teuthology.orchestra.run.smithi019.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-06T00:11:56.014 INFO:teuthology.orchestra.run.smithi059.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-06T00:11:56.018 INFO:teuthology.orchestra.run.smithi018.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-06T00:11:56.028 INFO:teuthology.orchestra.run.smithi018.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-06T00:11:56.028 INFO:teuthology.orchestra.run.smithi018.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-06T00:11:56.165 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:11:56.238 INFO:teuthology.orchestra.run.smithi019.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-06T00:11:56.278 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:11:56.477 INFO:teuthology.orchestra.run.smithi018.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-06T00:11:56.478 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:11:56.478 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:11:56.604 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:11:56.604 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:11:56.699 INFO:teuthology.orchestra.run.smithi018.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-06T00:11:56.748 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:11:56.962 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:57.068 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:11:57.068 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:11:57.112 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:57.578 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:58.247 INFO:teuthology.orchestra.run.smithi059.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-06T00:11:58.248 INFO:teuthology.orchestra.run.smithi059.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-06T00:11:58.409 INFO:teuthology.orchestra.run.smithi019.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-06T00:11:58.409 INFO:teuthology.orchestra.run.smithi019.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-06T00:11:58.414 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:58.576 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:58.858 INFO:teuthology.orchestra.run.smithi018.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-06T00:11:58.859 INFO:teuthology.orchestra.run.smithi018.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-06T00:11:59.054 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:59.790 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:11:59.919 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:12:00.456 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:12:01.254 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:12:01.379 INFO:teuthology.orchestra.run.smithi019.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-06T00:12:01.567 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:12:01.718 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-06T00:12:01.824 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:12:01.950 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-06T00:12:05.598 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:12:05.668 INFO:teuthology.orchestra.run.smithi019.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:12:06.225 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-06T00:12:06.295 INFO:teuthology.orchestra.run.smithi019.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-06T00:12:06.334 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:12:06.395 INFO:teuthology.orchestra.run.smithi019.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:12:06.522 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:12:06.537 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-06T00:12:06.537 INFO:teuthology.orchestra.run.smithi059.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:06.588 INFO:teuthology.orchestra.run.smithi019.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-06T00:12:06.588 INFO:teuthology.orchestra.run.smithi019.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:07.199 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-06T00:12:07.299 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:12:07.511 INFO:teuthology.orchestra.run.smithi018.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-06T00:12:07.511 INFO:teuthology.orchestra.run.smithi018.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:30.453 INFO:teuthology.orchestra.run.smithi019.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-06T00:12:30.454 INFO:teuthology.orchestra.run.smithi019.stdout:Sourcing file `/etc/default/grub' 2022-03-06T00:12:30.465 INFO:teuthology.orchestra.run.smithi019.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:12:30.491 INFO:teuthology.orchestra.run.smithi019.stdout:Generating grub configuration file ... 2022-03-06T00:12:30.771 INFO:teuthology.orchestra.run.smithi019.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:12:30.824 INFO:teuthology.orchestra.run.smithi019.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:31.121 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-06T00:12:31.121 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub' 2022-03-06T00:12:31.130 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:12:31.149 INFO:teuthology.orchestra.run.smithi059.stdout:Generating grub configuration file ... 2022-03-06T00:12:31.157 INFO:teuthology.orchestra.run.smithi018.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-06T00:12:31.158 INFO:teuthology.orchestra.run.smithi018.stdout:Sourcing file `/etc/default/grub' 2022-03-06T00:12:31.167 INFO:teuthology.orchestra.run.smithi018.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:12:31.189 INFO:teuthology.orchestra.run.smithi019.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:12:31.194 INFO:teuthology.orchestra.run.smithi018.stdout:Generating grub configuration file ... 2022-03-06T00:12:31.201 INFO:teuthology.orchestra.run.smithi019.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:12:31.455 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:12:31.483 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:31.526 INFO:teuthology.orchestra.run.smithi018.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:12:31.552 INFO:teuthology.orchestra.run.smithi018.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:31.842 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:12:31.859 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:12:31.926 INFO:teuthology.orchestra.run.smithi018.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:12:31.938 INFO:teuthology.orchestra.run.smithi018.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:12:33.815 INFO:teuthology.orchestra.run.smithi019.stdout:done 2022-03-06T00:12:34.152 DEBUG:teuthology.orchestra.run.smithi019:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:12:34.197 INFO:teuthology.orchestra.run.smithi019.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:12:34.197 INFO:teuthology.orchestra.run.smithi019.stdout:Status: install ok installed 2022-03-06T00:12:34.197 INFO:teuthology.orchestra.run.smithi019.stdout:Priority: optional 2022-03-06T00:12:34.198 INFO:teuthology.orchestra.run.smithi019.stdout:Section: kernel 2022-03-06T00:12:34.198 INFO:teuthology.orchestra.run.smithi019.stdout:Installed-Size: 19 2022-03-06T00:12:34.198 INFO:teuthology.orchestra.run.smithi019.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:12:34.199 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: amd64 2022-03-06T00:12:34.199 INFO:teuthology.orchestra.run.smithi019.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:12:34.199 INFO:teuthology.orchestra.run.smithi019.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:12:34.200 INFO:teuthology.orchestra.run.smithi019.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:12:34.200 INFO:teuthology.orchestra.run.smithi019.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:12:34.200 INFO:teuthology.orchestra.run.smithi019.stdout:Recommends: thermald 2022-03-06T00:12:34.200 INFO:teuthology.orchestra.run.smithi019.stdout:Description: Generic Linux kernel image 2022-03-06T00:12:34.201 INFO:teuthology.orchestra.run.smithi019.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:12:34.201 INFO:teuthology.orchestra.run.smithi019.stdout: available. 2022-03-06T00:12:34.201 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-06T00:12:34.202 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-06T00:12:34.202 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-06T00:12:34.202 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-06T00:12:34.203 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2022-03-06T00:12:34.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo apt-get clean 2022-03-06T00:12:34.304 DEBUG:teuthology.orchestra.run.smithi019:> sudo apt-get update 2022-03-06T00:12:34.508 INFO:teuthology.orchestra.run.smithi059.stdout:done 2022-03-06T00:12:34.553 INFO:teuthology.orchestra.run.smithi018.stdout:done 2022-03-06T00:12:34.569 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:12:34.569 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:12:34.641 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:12:34.723 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:12:34.864 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:12:34.891 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:12:34.891 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2022-03-06T00:12:34.892 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2022-03-06T00:12:34.892 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2022-03-06T00:12:34.892 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 19 2022-03-06T00:12:34.893 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:12:34.893 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2022-03-06T00:12:34.893 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:12:34.893 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:12:34.894 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:12:34.894 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:12:34.894 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2022-03-06T00:12:34.894 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2022-03-06T00:12:34.895 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:12:34.895 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2022-03-06T00:12:34.895 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-06T00:12:34.895 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-06T00:12:34.896 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-06T00:12:34.896 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-06T00:12:34.896 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2022-03-06T00:12:34.897 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2022-03-06T00:12:34.920 DEBUG:teuthology.orchestra.run.smithi018:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:12:34.959 INFO:teuthology.orchestra.run.smithi018.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:12:34.959 INFO:teuthology.orchestra.run.smithi018.stdout:Status: install ok installed 2022-03-06T00:12:34.959 INFO:teuthology.orchestra.run.smithi018.stdout:Priority: optional 2022-03-06T00:12:34.960 INFO:teuthology.orchestra.run.smithi018.stdout:Section: kernel 2022-03-06T00:12:34.960 INFO:teuthology.orchestra.run.smithi018.stdout:Installed-Size: 19 2022-03-06T00:12:34.960 INFO:teuthology.orchestra.run.smithi018.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:12:34.960 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: amd64 2022-03-06T00:12:34.961 INFO:teuthology.orchestra.run.smithi018.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:12:34.961 INFO:teuthology.orchestra.run.smithi018.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:12:34.961 INFO:teuthology.orchestra.run.smithi018.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:12:34.962 INFO:teuthology.orchestra.run.smithi018.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:12:34.962 INFO:teuthology.orchestra.run.smithi018.stdout:Recommends: thermald 2022-03-06T00:12:34.963 INFO:teuthology.orchestra.run.smithi018.stdout:Description: Generic Linux kernel image 2022-03-06T00:12:34.963 INFO:teuthology.orchestra.run.smithi018.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:12:34.963 INFO:teuthology.orchestra.run.smithi018.stdout: available. 2022-03-06T00:12:34.963 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-06T00:12:34.964 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-06T00:12:34.964 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-06T00:12:34.964 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-06T00:12:34.965 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi018.front.sepia.ceph.com, path=None, version=distro) 2022-03-06T00:12:34.965 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get clean 2022-03-06T00:12:34.966 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2022-03-06T00:12:35.038 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2022-03-06T00:12:35.239 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:12:35.239 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:12:35.306 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:12:35.308 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:12:35.309 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:12:35.376 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:12:35.391 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:12:35.458 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:12:36.142 INFO:teuthology.orchestra.run.smithi019.stdout:Reading package lists... 2022-03-06T00:12:36.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:12:36.236 INFO:teuthology.orchestra.run.smithi019.stdout:Reading package lists... 2022-03-06T00:12:36.414 INFO:teuthology.orchestra.run.smithi019.stdout:Building dependency tree... 2022-03-06T00:12:36.415 INFO:teuthology.orchestra.run.smithi019.stdout:Reading state information... 2022-03-06T00:12:36.593 INFO:teuthology.orchestra.run.smithi019.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-06T00:12:36.593 INFO:teuthology.orchestra.run.smithi019.stdout:0 upgraded, 0 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:12:36.595 DEBUG:teuthology.orchestra.run.smithi019:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:12:36.619 INFO:teuthology.orchestra.run.smithi019.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:12:36.619 INFO:teuthology.orchestra.run.smithi019.stdout:Status: install ok installed 2022-03-06T00:12:36.619 INFO:teuthology.orchestra.run.smithi019.stdout:Priority: optional 2022-03-06T00:12:36.619 INFO:teuthology.orchestra.run.smithi019.stdout:Section: kernel 2022-03-06T00:12:36.620 INFO:teuthology.orchestra.run.smithi019.stdout:Installed-Size: 19 2022-03-06T00:12:36.620 INFO:teuthology.orchestra.run.smithi019.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:12:36.620 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: amd64 2022-03-06T00:12:36.620 INFO:teuthology.orchestra.run.smithi019.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:12:36.621 INFO:teuthology.orchestra.run.smithi019.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:12:36.621 INFO:teuthology.orchestra.run.smithi019.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:12:36.621 INFO:teuthology.orchestra.run.smithi019.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:12:36.621 INFO:teuthology.orchestra.run.smithi019.stdout:Recommends: thermald 2022-03-06T00:12:36.621 INFO:teuthology.orchestra.run.smithi019.stdout:Description: Generic Linux kernel image 2022-03-06T00:12:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:12:36.622 INFO:teuthology.orchestra.run.smithi019.stdout: available. 2022-03-06T00:12:36.622 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2022-03-06T00:12:36.629 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.kl8HbgvbSw 2022-03-06T00:12:36.629 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub/grub.cfg /tmp/tmp.kl8HbgvbSw 2022-03-06T00:12:36.688 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.kl8HbgvbSw 2022-03-06T00:12:36.731 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.kl8HbgvbSw is 8KB 2022-03-06T00:12:36.746 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.kl8HbgvbSw 2022-03-06T00:12:36.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-06T00:12:36.809 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:12:36.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-06T00:12:36.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-06T00:12:36.844 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-03-06T00:12:36.854 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-06T00:12:36.855 DEBUG:teuthology.orchestra.run.smithi019:> sudo update-grub 2022-03-06T00:12:36.867 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:12:36.885 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-06T00:12:36.910 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:12:36.934 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-03-06T00:12:36.979 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-06T00:12:37.111 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-03-06T00:12:37.111 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-03-06T00:12:37.136 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-06T00:12:37.137 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-06T00:12:37.310 INFO:teuthology.orchestra.run.smithi059.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-06T00:12:37.311 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 0 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:12:37.311 INFO:teuthology.orchestra.run.smithi018.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-06T00:12:37.311 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 0 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:12:37.313 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:12:37.314 DEBUG:teuthology.orchestra.run.smithi018:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:12:37.336 INFO:teuthology.orchestra.run.smithi018.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:12:37.336 INFO:teuthology.orchestra.run.smithi018.stdout:Status: install ok installed 2022-03-06T00:12:37.336 INFO:teuthology.orchestra.run.smithi018.stdout:Priority: optional 2022-03-06T00:12:37.337 INFO:teuthology.orchestra.run.smithi018.stdout:Section: kernel 2022-03-06T00:12:37.337 INFO:teuthology.orchestra.run.smithi018.stdout:Installed-Size: 19 2022-03-06T00:12:37.337 INFO:teuthology.orchestra.run.smithi018.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:12:37.337 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: amd64 2022-03-06T00:12:37.338 INFO:teuthology.orchestra.run.smithi018.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:12:37.338 INFO:teuthology.orchestra.run.smithi018.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:12:37.338 INFO:teuthology.orchestra.run.smithi018.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:12:37.339 INFO:teuthology.orchestra.run.smithi018.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:12:37.339 INFO:teuthology.orchestra.run.smithi018.stdout:Recommends: thermald 2022-03-06T00:12:37.339 INFO:teuthology.orchestra.run.smithi018.stdout:Description: Generic Linux kernel image 2022-03-06T00:12:37.339 INFO:teuthology.orchestra.run.smithi018.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:12:37.340 INFO:teuthology.orchestra.run.smithi018.stdout: available. 2022-03-06T00:12:37.340 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2022-03-06T00:12:37.341 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:12:37.342 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2022-03-06T00:12:37.342 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2022-03-06T00:12:37.342 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2022-03-06T00:12:37.343 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 19 2022-03-06T00:12:37.343 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:12:37.343 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2022-03-06T00:12:37.343 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:12:37.344 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:12:37.344 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:12:37.344 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:12:37.344 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2022-03-06T00:12:37.345 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2022-03-06T00:12:37.345 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:12:37.345 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2022-03-06T00:12:37.345 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2022-03-06T00:12:37.350 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.CRqGhVvQiA 2022-03-06T00:12:37.350 DEBUG:teuthology.orchestra.run.smithi018:> sudo cp /boot/grub/grub.cfg /tmp/tmp.CRqGhVvQiA 2022-03-06T00:12:37.390 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.AFrHRhF8KA 2022-03-06T00:12:37.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub/grub.cfg /tmp/tmp.AFrHRhF8KA 2022-03-06T00:12:37.406 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.AFrHRhF8KA 2022-03-06T00:12:37.409 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.CRqGhVvQiA 2022-03-06T00:12:37.444 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.CRqGhVvQiA is 8KB 2022-03-06T00:12:37.448 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.AFrHRhF8KA is 8KB 2022-03-06T00:12:37.459 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.CRqGhVvQiA 2022-03-06T00:12:37.464 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.AFrHRhF8KA 2022-03-06T00:12:37.466 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-06T00:12:37.472 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-06T00:12:37.527 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:12:37.528 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-06T00:12:37.528 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-06T00:12:37.530 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:12:37.530 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-06T00:12:37.530 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-06T00:12:37.564 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-06T00:12:37.565 DEBUG:teuthology.orchestra.run.smithi018:> sudo update-grub 2022-03-06T00:12:37.581 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-06T00:12:37.581 DEBUG:teuthology.orchestra.run.smithi059:> sudo update-grub 2022-03-06T00:12:38.027 INFO:teuthology.orchestra.run.smithi019.stderr:Sourcing file `/etc/default/grub' 2022-03-06T00:12:38.034 INFO:teuthology.orchestra.run.smithi019.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:12:38.047 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2022-03-06T00:12:38.307 INFO:teuthology.orchestra.run.smithi019.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:12:38.319 INFO:teuthology.orchestra.run.smithi019.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:38.770 INFO:teuthology.orchestra.run.smithi019.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:12:38.785 INFO:teuthology.orchestra.run.smithi019.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:12:38.792 INFO:teuthology.orchestra.run.smithi018.stderr:Sourcing file `/etc/default/grub' 2022-03-06T00:12:38.794 INFO:teuthology.orchestra.run.smithi018.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:12:38.801 INFO:teuthology.orchestra.run.smithi018.stderr:Generating grub configuration file ... 2022-03-06T00:12:38.841 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub' 2022-03-06T00:12:38.842 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:12:38.857 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2022-03-06T00:12:39.053 INFO:teuthology.orchestra.run.smithi018.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:12:39.064 INFO:teuthology.orchestra.run.smithi018.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:39.103 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:12:39.115 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:12:39.459 INFO:teuthology.orchestra.run.smithi018.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:12:39.470 INFO:teuthology.orchestra.run.smithi018.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:12:39.488 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:12:39.500 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:12:40.455 INFO:teuthology.orchestra.run.smithi019.stderr:done 2022-03-06T00:12:40.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2022-03-06T00:12:41.057 INFO:teuthology.orchestra.run.smithi018.stderr:done 2022-03-06T00:12:41.059 DEBUG:teuthology.orchestra.run.smithi018:> sudo shutdown -r now 2022-03-06T00:12:41.137 INFO:teuthology.orchestra.run.smithi059.stderr:done 2022-03-06T00:12:41.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2022-03-06T00:13:10.490 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-06T00:13:10.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-03-06T00:13:10.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:13:11.066 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-06T00:13:11.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-06T00:13:11.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:13:11.142 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-06T00:13:11.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-03-06T00:13:11.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:13:28.907 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2022-03-06T00:13:29.419 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.18 2022-03-06T00:13:29.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2022-03-06T00:13:58.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-03-06T00:13:58.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:13:59.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-06T00:13:59.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:13:59.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-03-06T00:13:59.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:14:01.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2022-03-06T00:14:02.479 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.18 2022-03-06T00:14:02.731 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2022-03-06T00:14:32.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-03-06T00:14:32.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:14:32.241 DEBUG:teuthology.orchestra.run.smithi019:> true 2022-03-06T00:14:32.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-06T00:14:32.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:14:32.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-03-06T00:14:33.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:14:33.157 DEBUG:teuthology.orchestra.run.smithi018:> true 2022-03-06T00:14:33.171 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-03-06T00:14:33.172 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-30-generic"... 2022-03-06T00:14:33.172 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2022-03-06T00:14:33.178 INFO:teuthology.orchestra.run.smithi019.stdout:5.13.0-30-generic 2022-03-06T00:14:33.178 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-06T00:14:33.179 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-06T00:14:33.179 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-06T00:14:33.228 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-06T00:14:33.228 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-03-06T00:14:33.228 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2022-03-06T00:14:33.236 INFO:teuthology.orchestra.run.smithi018.stdout:5.13.0-30-generic 2022-03-06T00:14:33.236 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-06T00:14:33.237 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-06T00:14:33.237 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-06T00:14:33.354 DEBUG:teuthology.orchestra.run.smithi059:> true 2022-03-06T00:14:33.843 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-03-06T00:14:33.844 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-30-generic"... 2022-03-06T00:14:33.844 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2022-03-06T00:14:33.849 INFO:teuthology.orchestra.run.smithi059.stdout:5.13.0-30-generic 2022-03-06T00:14:33.849 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-06T00:14:33.849 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-06T00:14:33.850 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-06T00:14:34.181 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-06T00:14:34.182 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-06T00:14:34.238 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-06T00:14:34.239 DEBUG:teuthology.orchestra.run.smithi018:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-06T00:14:34.261 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2022-03-06T00:14:34.284 DEBUG:teuthology.parallel:result is None 2022-03-06T00:14:34.295 INFO:teuthology.orchestra.run.smithi018.stdout:ttyS1 2022-03-06T00:14:34.324 DEBUG:teuthology.parallel:result is None 2022-03-06T00:14:34.851 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-06T00:14:34.852 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-06T00:14:34.921 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2022-03-06T00:14:34.953 DEBUG:teuthology.parallel:result is None 2022-03-06T00:14:34.953 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-06T00:14:34.974 INFO:teuthology.task.internal:Creating test directory... 2022-03-06T00:14:34.975 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-06T00:14:34.978 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-06T00:14:34.982 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-06T00:14:34.991 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-06T00:14:35.131 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-06T00:14:35.303 INFO:teuthology.task.internal:Creating archive directory... 2022-03-06T00:14:35.304 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-06T00:14:35.307 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-06T00:14:35.312 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-06T00:14:35.341 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-06T00:14:35.465 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-06T00:14:35.466 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-06T00:14:35.471 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-06T00:14:35.475 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-06T00:14:35.496 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:14:35.504 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:14:35.505 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:14:35.506 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:14:35.515 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:14:35.517 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:14:35.519 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-06T00:14:35.571 INFO:teuthology.task.internal:Configuring sudo... 2022-03-06T00:14:35.572 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-06T00:14:35.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-06T00:14:35.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-06T00:14:35.594 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-06T00:14:35.769 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-06T00:14:35.770 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-06T00:14:35.772 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-06T00:14:35.775 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-06T00:14:35.782 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-06T00:14:35.788 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-06T00:14:35.794 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:14:35.795 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-06T00:14:35.859 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-06T00:14:35.867 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-06T00:14:35.875 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:14:35.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-06T00:14:35.898 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-06T00:14:35.908 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-06T00:14:35.958 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:14:35.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-06T00:14:35.979 DEBUG:teuthology.orchestra.run.smithi018:> sudo service rsyslog restart 2022-03-06T00:14:35.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2022-03-06T00:14:35.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2022-03-06T00:14:36.033 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-06T00:14:36.050 INFO:teuthology.task.internal:Starting timer... 2022-03-06T00:14:36.051 INFO:teuthology.run_tasks:Running task pcp... 2022-03-06T00:14:36.160 INFO:teuthology.run_tasks:Running task selinux... 2022-03-06T00:14:36.237 DEBUG:teuthology.task.selinux:Excluding smithi018: OS 'ubuntu' does not support SELinux 2022-03-06T00:14:36.239 DEBUG:teuthology.task.selinux:Excluding smithi019: OS 'ubuntu' does not support SELinux 2022-03-06T00:14:36.239 DEBUG:teuthology.task.selinux:Excluding smithi059: OS 'ubuntu' does not support SELinux 2022-03-06T00:14:36.240 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-06T00:14:36.241 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-06T00:14:36.241 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-06T00:14:36.242 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-06T00:14:36.323 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-06T00:14:36.323 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-06T00:14:36.347 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-06T00:14:36.349 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi018.front.sepia.ceph.com,smithi019.front.sepia.ceph.com,smithi059.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-06T00:17:45.244 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi018.front.sepia.ceph.com'), Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi059.front.sepia.ceph.com')] 2022-03-06T00:17:45.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-03-06T00:17:45.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:17:45.303 DEBUG:teuthology.orchestra.run.smithi018:> true 2022-03-06T00:17:45.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2022-03-06T00:17:45.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:17:45.484 DEBUG:teuthology.orchestra.run.smithi019:> true 2022-03-06T00:17:45.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-03-06T00:17:45.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:17:45.657 DEBUG:teuthology.orchestra.run.smithi059:> true 2022-03-06T00:17:45.780 INFO:teuthology.run_tasks:Running task clock... 2022-03-06T00:17:45.796 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-06T00:17:45.796 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-06T00:17:45.798 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-06T00:17:45.801 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-06T00:17:45.801 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-06T00:17:45.804 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-06T00:17:45.804 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-06T00:17:45.834 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-06T00:17:45.834 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: Command line: ntpd -gq 2022-03-06T00:17:45.835 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: proto: precision = 0.056 usec (-24) 2022-03-06T00:17:45.835 INFO:teuthology.orchestra.run.smithi018.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:17:45.836 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:17:45.836 INFO:teuthology.orchestra.run.smithi018.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:17:45.837 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:17:45.838 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: Listen and drop on 0 v6wildcard [::]:123 2022-03-06T00:17:45.838 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-06T00:17:45.838 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: Listen normally on 2 lo 127.0.0.1:123 2022-03-06T00:17:45.838 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: Listen normally on 3 ens1f0 172.21.15.18:123 2022-03-06T00:17:45.839 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: Listen normally on 4 lo [::1]:123 2022-03-06T00:17:45.839 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1362%4]:123 2022-03-06T00:17:45.839 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:45 ntpd[8921]: Listening on routing socket on fd #22 for interface updates 2022-03-06T00:17:45.840 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-06T00:17:45.840 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: Command line: ntpd -gq 2022-03-06T00:17:45.840 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: proto: precision = 0.055 usec (-24) 2022-03-06T00:17:45.841 INFO:teuthology.orchestra.run.smithi019.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:17:45.842 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:17:45.842 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:17:45.842 INFO:teuthology.orchestra.run.smithi019.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:17:45.843 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: Listen and drop on 0 v6wildcard [::]:123 2022-03-06T00:17:45.844 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-06T00:17:45.844 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: Listen normally on 2 lo 127.0.0.1:123 2022-03-06T00:17:45.844 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: Listen normally on 3 ens1f0 172.21.15.19:123 2022-03-06T00:17:45.844 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: Listen normally on 4 lo [::1]:123 2022-03-06T00:17:45.845 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:161e%4]:123 2022-03-06T00:17:45.845 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:45 ntpd[8982]: Listening on routing socket on fd #22 for interface updates 2022-03-06T00:17:45.857 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-06T00:17:45.858 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: Command line: ntpd -gq 2022-03-06T00:17:45.858 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: proto: precision = 0.048 usec (-24) 2022-03-06T00:17:45.859 INFO:teuthology.orchestra.run.smithi059.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:17:45.859 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:17:45.860 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:17:45.860 INFO:teuthology.orchestra.run.smithi059.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:17:45.861 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: Listen and drop on 0 v6wildcard [::]:123 2022-03-06T00:17:45.861 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-06T00:17:45.862 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: Listen normally on 2 lo 127.0.0.1:123 2022-03-06T00:17:45.862 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: Listen normally on 3 ens1f0 172.21.15.59:123 2022-03-06T00:17:45.862 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: Listen normally on 4 lo [::1]:123 2022-03-06T00:17:45.862 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:192a%4]:123 2022-03-06T00:17:45.863 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:45 ntpd[8995]: Listening on routing socket on fd #22 for interface updates 2022-03-06T00:17:46.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:46.840 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:46.840 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.841 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.841 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:46.841 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:46.842 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.842 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.843 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:46.843 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:46 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:46.845 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:46 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.846 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:46 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.846 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:46 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.847 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:46 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.847 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:46 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:46.848 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:46 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:46.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:46 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:46 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:46 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:46.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:46 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:46.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:46 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:46.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:46 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:47.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:47 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:47.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:47 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:47.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:47 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:47.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:47 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:47.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:47 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:47.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:47 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:47.858 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:47 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:47.858 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:47 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:47.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:47 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:47.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:47 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:47.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:47 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:47.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:47 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:48.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:48.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:48.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:48.840 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:48.840 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.841 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.841 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:48.842 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:48 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:48.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:48 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:48 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.845 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:48 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.845 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:48 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.846 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:48 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:48.846 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:48 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:48.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:48.862 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:48.862 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:48.862 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:48.863 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:48.863 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:48 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:49.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:49 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:49 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:49 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:49 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:49 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:49.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:49 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:49.838 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.838 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.839 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.839 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.839 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.839 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.840 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:49.840 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:49.840 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:49.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:49 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:49.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:49.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:49.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:49.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:49.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:49.862 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:49 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:50.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:50.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:50.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:50.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:50.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.840 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.840 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:50.840 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:50 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:50.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.843 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.843 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.843 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.843 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:50.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:50.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:50.845 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:50 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:50.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:50.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:50.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:50.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:50.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:50.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:50 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:51.834 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:51 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:51 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:51 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:51 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:51 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:51.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:51 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:51.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.843 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.843 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:51.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:51.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.845 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.845 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:51.846 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:51 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:51.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.862 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.862 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:51.862 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:51.863 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:51.863 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:51.864 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:51.864 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:51 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:52.835 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.836 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.837 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:52.838 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:52.838 INFO:teuthology.orchestra.run.smithi018.stdout: 6 Mar 00:17:52 ntpd[8921]: ntpd: time slew -0.010095 s 2022-03-06T00:17:52.839 INFO:teuthology.orchestra.run.smithi018.stdout:ntpd: time slew -0.010095s 2022-03-06T00:17:52.839 INFO:teuthology.orchestra.run.smithi018.stderr: 6 Mar 00:17:52 ntpd[8921]: can't open /var/log/ntpstats/loopstats.20220306: Permission denied 2022-03-06T00:17:52.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.841 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:52.842 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:52.843 INFO:teuthology.orchestra.run.smithi019.stdout: 6 Mar 00:17:52 ntpd[8982]: ntpd: time slew -0.000286 s 2022-03-06T00:17:52.843 INFO:teuthology.orchestra.run.smithi019.stdout:ntpd: time slew -0.000286s 2022-03-06T00:17:52.844 INFO:teuthology.orchestra.run.smithi019.stderr: 6 Mar 00:17:52 ntpd[8982]: can't open /var/log/ntpstats/loopstats.20220306: Permission denied 2022-03-06T00:17:52.858 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.858 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:17:52.859 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:17:52.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:17:52.860 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:17:52.860 INFO:teuthology.orchestra.run.smithi059.stdout: 6 Mar 00:17:52 ntpd[8995]: ntpd: time slew -0.000343 s 2022-03-06T00:17:52.861 INFO:teuthology.orchestra.run.smithi059.stdout:ntpd: time slew -0.000343s 2022-03-06T00:17:52.861 INFO:teuthology.orchestra.run.smithi059.stderr: 6 Mar 00:17:52 ntpd[8995]: can't open /var/log/ntpstats/loopstats.20220306: Permission denied 2022-03-06T00:17:52.885 INFO:teuthology.orchestra.run.smithi018.stdout: remote refid st t when poll reach delay offset jitter 2022-03-06T00:17:52.885 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================== 2022-03-06T00:17:52.885 INFO:teuthology.orchestra.run.smithi018.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.886 INFO:teuthology.orchestra.run.smithi018.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.886 INFO:teuthology.orchestra.run.smithi018.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.886 INFO:teuthology.orchestra.run.smithi018.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.895 INFO:teuthology.orchestra.run.smithi019.stdout: remote refid st t when poll reach delay offset jitter 2022-03-06T00:17:52.895 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================== 2022-03-06T00:17:52.895 INFO:teuthology.orchestra.run.smithi019.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.896 INFO:teuthology.orchestra.run.smithi019.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.896 INFO:teuthology.orchestra.run.smithi019.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.896 INFO:teuthology.orchestra.run.smithi019.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.924 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2022-03-06T00:17:52.924 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2022-03-06T00:17:52.924 INFO:teuthology.orchestra.run.smithi059.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.925 INFO:teuthology.orchestra.run.smithi059.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.925 INFO:teuthology.orchestra.run.smithi059.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.925 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:17:52.926 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-06T00:17:52.940 INFO:tasks.kubeadm:Kubeadm start 2022-03-06T00:17:52.940 INFO:tasks.kubeadm:Config: {'version': '1.21'} 2022-03-06T00:17:52.941 DEBUG:teuthology.orchestra.run.smithi018:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-06T00:17:52.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-06T00:17:52.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-06T00:17:52.968 INFO:teuthology.orchestra.run.smithi018.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-06T00:17:52.971 INFO:teuthology.orchestra.run.smithi019.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-06T00:17:52.972 INFO:teuthology.orchestra.run.smithi059.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-06T00:17:52.975 INFO:teuthology.orchestra.run.smithi018.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-06T00:17:52.980 INFO:teuthology.orchestra.run.smithi059.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-06T00:17:52.981 INFO:teuthology.orchestra.run.smithi019.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-06T00:17:52.982 INFO:teuthology.orchestra.run.smithi018.stdout:net.ipv4.ip_forward = 1 2022-03-06T00:17:52.990 INFO:teuthology.orchestra.run.smithi019.stdout:net.ipv4.ip_forward = 1 2022-03-06T00:17:52.991 INFO:teuthology.orchestra.run.smithi059.stdout:net.ipv4.ip_forward = 1 2022-03-06T00:17:53.058 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:17:53.058 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-06T00:17:53.074 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:17:53.074 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/docker/daemon.json 2022-03-06T00:17:53.135 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:17:53.136 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-06T00:17:53.152 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:17:53.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/docker/daemon.json 2022-03-06T00:17:53.166 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:17:53.166 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-06T00:17:53.183 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:17:53.184 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/docker/daemon.json 2022-03-06T00:17:53.240 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl restart docker || true 2022-03-06T00:17:53.243 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl restart docker || true 2022-03-06T00:17:53.245 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl restart docker || true 2022-03-06T00:17:54.401 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-06T00:17:54.408 INFO:tasks.kubeadm:172.21.15.18 is in 172.21.0.0/20 at pos 3857 2022-03-06T00:17:54.423 INFO:tasks.kubeadm:Pod subnet: 10.248.136.0/21 2022-03-06T00:17:54.423 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-06T00:17:54.423 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-06T00:17:54.426 DEBUG:teuthology.orchestra.run.smithi019:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-06T00:17:54.429 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-06T00:17:54.449 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-06T00:17:54.449 INFO:teuthology.orchestra.run.smithi018.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:54.450 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-06T00:17:54.457 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-03-06T00:17:54.457 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:54.457 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-03-06T00:17:54.458 INFO:teuthology.orchestra.run.smithi019.stderr: 2022-03-06T00:17:54.458 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:54.458 INFO:teuthology.orchestra.run.smithi019.stderr: 2022-03-06T00:17:54.634 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:17:54.635 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:17:54.716 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:17:54.717 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:17:54.723 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:17:54.742 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:17:54.802 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:17:54.806 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:17:54.807 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:17:54.893 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:17:54.897 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:17:54.897 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:17:56.137 INFO:teuthology.orchestra.run.smithi019.stdout:Reading package lists... 2022-03-06T00:17:56.144 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-03-06T00:17:56.233 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-06T00:17:56.314 INFO:teuthology.orchestra.run.smithi019.stdout:Building dependency tree... 2022-03-06T00:17:56.315 INFO:teuthology.orchestra.run.smithi019.stdout:Reading state information... 2022-03-06T00:17:56.318 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-03-06T00:17:56.318 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-03-06T00:17:56.409 INFO:teuthology.orchestra.run.smithi019.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:17:56.412 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-06T00:17:56.413 INFO:teuthology.orchestra.run.smithi059.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:17:56.413 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-06T00:17:56.424 INFO:teuthology.orchestra.run.smithi019.stderr: 2022-03-06T00:17:56.424 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:56.424 INFO:teuthology.orchestra.run.smithi019.stderr: 2022-03-06T00:17:56.429 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-03-06T00:17:56.430 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:56.430 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-03-06T00:17:56.481 INFO:teuthology.orchestra.run.smithi019.stdout:Reading package lists... 2022-03-06T00:17:56.483 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-03-06T00:17:56.503 INFO:teuthology.orchestra.run.smithi018.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:17:56.517 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-06T00:17:56.517 INFO:teuthology.orchestra.run.smithi018.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:56.517 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-06T00:17:56.572 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-06T00:17:56.660 INFO:teuthology.orchestra.run.smithi019.stdout:Building dependency tree... 2022-03-06T00:17:56.660 INFO:teuthology.orchestra.run.smithi019.stdout:Reading state information... 2022-03-06T00:17:56.661 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-03-06T00:17:56.662 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-03-06T00:17:56.750 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-06T00:17:56.750 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-06T00:17:56.836 INFO:teuthology.orchestra.run.smithi019.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-06T00:17:56.837 INFO:teuthology.orchestra.run.smithi019.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-06T00:17:56.837 INFO:teuthology.orchestra.run.smithi019.stdout:The following NEW packages will be installed: 2022-03-06T00:17:56.838 INFO:teuthology.orchestra.run.smithi019.stdout: apt-transport-https 2022-03-06T00:17:56.845 INFO:teuthology.orchestra.run.smithi059.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-06T00:17:56.846 INFO:teuthology.orchestra.run.smithi059.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-06T00:17:56.846 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2022-03-06T00:17:56.846 INFO:teuthology.orchestra.run.smithi059.stdout: apt-transport-https 2022-03-06T00:17:56.938 INFO:teuthology.orchestra.run.smithi018.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-06T00:17:56.939 INFO:teuthology.orchestra.run.smithi018.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-06T00:17:56.939 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2022-03-06T00:17:56.939 INFO:teuthology.orchestra.run.smithi018.stdout: apt-transport-https 2022-03-06T00:17:57.060 INFO:teuthology.orchestra.run.smithi019.stdout:0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:17:57.061 INFO:teuthology.orchestra.run.smithi019.stdout:Need to get 4,680 B of archives. 2022-03-06T00:17:57.061 INFO:teuthology.orchestra.run.smithi019.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-06T00:17:57.061 INFO:teuthology.orchestra.run.smithi019.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-06T00:17:57.068 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:17:57.068 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 4,680 B of archives. 2022-03-06T00:17:57.068 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-06T00:17:57.068 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-06T00:17:57.164 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:17:57.164 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 4,680 B of archives. 2022-03-06T00:17:57.164 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-06T00:17:57.165 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-06T00:17:57.297 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:17:57.297 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:17:57.297 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:17:57.298 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:17:57.298 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:17:57.299 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:17:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:17:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:17:57.315 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:17:57.315 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:17:57.316 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:17:57.316 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:17:57.320 INFO:teuthology.orchestra.run.smithi019.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:17:57.321 INFO:teuthology.orchestra.run.smithi059.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:17:57.367 INFO:teuthology.orchestra.run.smithi019.stdout:Fetched 4,680 B in 0s (21.6 kB/s) 2022-03-06T00:17:57.371 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 4,680 B in 0s (21.9 kB/s) 2022-03-06T00:17:57.392 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:17:57.392 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:17:57.392 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:17:57.412 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:17:57.413 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:17:57.413 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:17:57.418 INFO:teuthology.orchestra.run.smithi018.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:17:57.440 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package apt-transport-https. 2022-03-06T00:17:57.456 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package apt-transport-https. 2022-03-06T00:17:57.463 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 4,680 B in 0s (21.5 kB/s) 2022-03-06T00:17:57.482 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-06T00:17:57.485 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-06T00:17:57.499 INFO:teuthology.orchestra.run.smithi019.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-06T00:17:57.503 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-06T00:17:57.514 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-06T00:17:57.536 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package apt-transport-https. 2022-03-06T00:17:57.579 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-06T00:17:57.579 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-06T00:17:57.582 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-06T00:17:57.626 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-06T00:17:57.784 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-06T00:17:57.883 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-06T00:17:58.024 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-06T00:17:58.324 INFO:teuthology.orchestra.run.smithi059.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-06T00:17:58.336 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-03-06T00:17:58.336 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:58.337 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-03-06T00:17:58.347 INFO:teuthology.orchestra.run.smithi019.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-06T00:17:58.360 INFO:teuthology.orchestra.run.smithi019.stderr: 2022-03-06T00:17:58.360 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:58.361 INFO:teuthology.orchestra.run.smithi019.stderr: 2022-03-06T00:17:58.535 INFO:teuthology.orchestra.run.smithi018.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-06T00:17:58.551 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-06T00:17:58.551 INFO:teuthology.orchestra.run.smithi018.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:17:58.551 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-06T00:17:58.592 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:17:58.592 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:17:58.611 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:17:58.612 INFO:teuthology.orchestra.run.smithi019.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:17:58.662 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:17:58.682 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:17:58.685 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:17:58.744 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:17:58.759 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-06T00:17:58.764 INFO:teuthology.orchestra.run.smithi019.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:17:58.796 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:17:58.878 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:17:58.923 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-06T00:17:58.943 INFO:teuthology.orchestra.run.smithi019.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-06T00:17:58.960 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:17:59.221 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-06T00:17:59.241 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-06T00:17:59.256 INFO:teuthology.orchestra.run.smithi019.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:17:59.278 INFO:teuthology.orchestra.run.smithi019.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-06T00:17:59.366 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 63.3 kB in 1s (72.0 kB/s) 2022-03-06T00:17:59.423 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 63.3 kB in 1s (88.0 kB/s) 2022-03-06T00:17:59.637 INFO:teuthology.orchestra.run.smithi019.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:17:59.880 INFO:teuthology.orchestra.run.smithi019.stdout:Fetched 2,133 kB in 1s (1,550 kB/s) 2022-03-06T00:18:00.251 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-03-06T00:18:00.340 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-06T00:18:00.431 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-03-06T00:18:00.432 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-03-06T00:18:00.515 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-06T00:18:00.516 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-06T00:18:00.528 INFO:teuthology.orchestra.run.smithi059.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:18:00.545 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-03-06T00:18:00.546 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:18:00.546 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-03-06T00:18:00.603 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-03-06T00:18:00.610 INFO:teuthology.orchestra.run.smithi018.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:18:00.622 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-06T00:18:00.622 INFO:teuthology.orchestra.run.smithi018.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:18:00.623 INFO:teuthology.orchestra.run.smithi018.stderr: 2022-03-06T00:18:00.678 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2022-03-06T00:18:00.754 INFO:teuthology.orchestra.run.smithi019.stdout:Reading package lists... 2022-03-06T00:18:00.765 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-03-06T00:18:00.766 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-03-06T00:18:00.840 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2022-03-06T00:18:00.841 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2022-03-06T00:18:00.902 INFO:teuthology.orchestra.run.smithi059.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-06T00:18:00.903 INFO:teuthology.orchestra.run.smithi059.stdout:bridge-utils set to manually installed. 2022-03-06T00:18:00.903 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2022-03-06T00:18:00.903 INFO:teuthology.orchestra.run.smithi059.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-06T00:18:00.904 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2022-03-06T00:18:00.904 INFO:teuthology.orchestra.run.smithi059.stdout: nftables 2022-03-06T00:18:00.934 INFO:teuthology.orchestra.run.smithi019.stdout:Building dependency tree... 2022-03-06T00:18:00.935 INFO:teuthology.orchestra.run.smithi019.stdout:Reading state information... 2022-03-06T00:18:00.954 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2022-03-06T00:18:00.956 INFO:teuthology.orchestra.run.smithi059.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-06T00:18:00.978 INFO:teuthology.orchestra.run.smithi018.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-06T00:18:00.978 INFO:teuthology.orchestra.run.smithi018.stdout:bridge-utils set to manually installed. 2022-03-06T00:18:00.979 INFO:teuthology.orchestra.run.smithi018.stdout:The following additional packages will be installed: 2022-03-06T00:18:00.979 INFO:teuthology.orchestra.run.smithi018.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-06T00:18:00.980 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2022-03-06T00:18:00.980 INFO:teuthology.orchestra.run.smithi018.stdout: nftables 2022-03-06T00:18:01.026 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2022-03-06T00:18:01.027 INFO:teuthology.orchestra.run.smithi018.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-06T00:18:01.030 INFO:teuthology.orchestra.run.smithi019.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:18:01.048 INFO:teuthology.orchestra.run.smithi019.stderr: 2022-03-06T00:18:01.048 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:18:01.049 INFO:teuthology.orchestra.run.smithi019.stderr: 2022-03-06T00:18:01.102 INFO:teuthology.orchestra.run.smithi019.stdout:Reading package lists... 2022-03-06T00:18:01.148 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 8 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:18:01.149 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 73.6 MB of archives. 2022-03-06T00:18:01.149 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-06T00:18:01.149 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-06T00:18:01.195 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-06T00:18:01.220 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 8 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:18:01.220 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 73.6 MB of archives. 2022-03-06T00:18:01.220 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-06T00:18:01.220 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-06T00:18:01.249 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-06T00:18:01.285 INFO:teuthology.orchestra.run.smithi019.stdout:Building dependency tree... 2022-03-06T00:18:01.286 INFO:teuthology.orchestra.run.smithi019.stdout:Reading state information... 2022-03-06T00:18:01.332 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-06T00:18:01.396 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-06T00:18:01.441 INFO:teuthology.orchestra.run.smithi019.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-06T00:18:01.442 INFO:teuthology.orchestra.run.smithi019.stdout:bridge-utils set to manually installed. 2022-03-06T00:18:01.442 INFO:teuthology.orchestra.run.smithi019.stdout:The following additional packages will be installed: 2022-03-06T00:18:01.442 INFO:teuthology.orchestra.run.smithi019.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-06T00:18:01.443 INFO:teuthology.orchestra.run.smithi019.stdout:Suggested packages: 2022-03-06T00:18:01.443 INFO:teuthology.orchestra.run.smithi019.stdout: nftables 2022-03-06T00:18:01.455 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-06T00:18:01.492 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-06T00:18:01.502 INFO:teuthology.orchestra.run.smithi019.stdout:The following NEW packages will be installed: 2022-03-06T00:18:01.502 INFO:teuthology.orchestra.run.smithi019.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-06T00:18:01.522 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-06T00:18:01.556 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-06T00:18:01.779 INFO:teuthology.orchestra.run.smithi019.stdout:0 upgraded, 8 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:18:01.780 INFO:teuthology.orchestra.run.smithi019.stdout:Need to get 73.6 MB of archives. 2022-03-06T00:18:01.780 INFO:teuthology.orchestra.run.smithi019.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-06T00:18:01.780 INFO:teuthology.orchestra.run.smithi019.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-06T00:18:01.948 INFO:teuthology.orchestra.run.smithi019.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-06T00:18:02.002 INFO:teuthology.orchestra.run.smithi019.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-06T00:18:02.090 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-06T00:18:02.135 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-06T00:18:02.172 INFO:teuthology.orchestra.run.smithi019.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-06T00:18:02.278 INFO:teuthology.orchestra.run.smithi019.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-06T00:18:02.813 INFO:teuthology.orchestra.run.smithi019.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-06T00:18:03.113 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-06T00:18:03.121 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-06T00:18:03.295 INFO:teuthology.orchestra.run.smithi019.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-06T00:18:03.416 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-06T00:18:03.454 INFO:teuthology.orchestra.run.smithi019.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-06T00:18:03.483 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-06T00:18:03.853 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:18:03.853 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:18:03.853 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:18:03.862 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:18:03.863 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:18:03.863 INFO:teuthology.orchestra.run.smithi019.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:18:03.868 INFO:teuthology.orchestra.run.smithi019.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:18:03.920 INFO:teuthology.orchestra.run.smithi019.stdout:Fetched 73.6 MB in 2s (34.8 MB/s) 2022-03-06T00:18:03.997 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:18:03.997 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:18:03.997 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:18:04.005 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:18:04.005 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:18:04.006 INFO:teuthology.orchestra.run.smithi018.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:18:04.011 INFO:teuthology.orchestra.run.smithi018.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:18:04.025 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:18:04.025 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:18:04.026 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:18:04.035 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:18:04.035 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:18:04.035 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:18:04.041 INFO:teuthology.orchestra.run.smithi059.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:18:04.060 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 73.6 MB in 3s (26.9 MB/s) 2022-03-06T00:18:04.097 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 73.6 MB in 3s (26.0 MB/s) 2022-03-06T00:18:04.137 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package conntrack. 2022-03-06T00:18:04.152 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package conntrack. 2022-03-06T00:18:04.154 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package conntrack. 2022-03-06T00:18:04.183 INFO:teuthology.orchestra.run.smithi019.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-06T00:18:04.186 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-06T00:18:04.196 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-06T00:18:04.197 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-06T00:18:04.200 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-06T00:18:04.200 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-06T00:18:04.235 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-06T00:18:04.236 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-06T00:18:04.242 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-06T00:18:04.512 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package cri-tools. 2022-03-06T00:18:04.524 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-06T00:18:04.538 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package cri-tools. 2022-03-06T00:18:04.546 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-06T00:18:04.561 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-06T00:18:04.605 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-06T00:18:04.680 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cri-tools. 2022-03-06T00:18:04.694 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-06T00:18:04.738 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-06T00:18:05.714 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ebtables. 2022-03-06T00:18:05.724 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ebtables. 2022-03-06T00:18:05.729 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-06T00:18:05.737 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-06T00:18:05.790 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-06T00:18:05.791 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-06T00:18:05.841 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package ebtables. 2022-03-06T00:18:05.856 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-06T00:18:05.914 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-06T00:18:06.034 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-06T00:18:06.048 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-06T00:18:06.060 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-06T00:18:06.074 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-06T00:18:06.084 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-06T00:18:06.119 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-06T00:18:06.186 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-06T00:18:06.199 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-06T00:18:06.227 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-06T00:18:07.537 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package socat. 2022-03-06T00:18:07.549 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-06T00:18:07.563 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package socat. 2022-03-06T00:18:07.577 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-06T00:18:07.586 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-06T00:18:07.613 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-06T00:18:07.696 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package socat. 2022-03-06T00:18:07.709 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-06T00:18:07.769 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-06T00:18:07.957 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package kubelet. 2022-03-06T00:18:07.970 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-06T00:18:08.006 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-06T00:18:08.007 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package kubelet. 2022-03-06T00:18:08.022 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-06T00:18:08.064 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-06T00:18:08.164 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package kubelet. 2022-03-06T00:18:08.178 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-06T00:18:08.222 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-06T00:18:10.442 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package kubectl. 2022-03-06T00:18:10.457 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-06T00:18:10.457 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package kubectl. 2022-03-06T00:18:10.467 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-06T00:18:10.500 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-06T00:18:10.507 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-06T00:18:10.783 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package kubectl. 2022-03-06T00:18:10.793 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-06T00:18:10.824 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-06T00:18:11.669 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package kubeadm. 2022-03-06T00:18:11.681 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-06T00:18:11.693 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package kubeadm. 2022-03-06T00:18:11.706 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-06T00:18:11.719 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-06T00:18:11.751 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-06T00:18:12.185 INFO:teuthology.orchestra.run.smithi019.stdout:Selecting previously unselected package kubeadm. 2022-03-06T00:18:12.200 INFO:teuthology.orchestra.run.smithi019.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-06T00:18:12.243 INFO:teuthology.orchestra.run.smithi019.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-06T00:18:12.852 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-06T00:18:12.975 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-06T00:18:12.989 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-06T00:18:13.088 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-06T00:18:13.101 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-06T00:18:13.258 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-06T00:18:13.434 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-06T00:18:13.649 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-06T00:18:13.794 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-06T00:18:13.942 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-06T00:18:14.068 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-06T00:18:14.095 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-06T00:18:14.214 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-06T00:18:14.235 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-06T00:18:14.342 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-06T00:18:14.469 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-06T00:18:14.486 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-06T00:18:14.648 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-06T00:18:14.697 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-06T00:18:14.718 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-06T00:18:14.766 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-06T00:18:14.900 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-06T00:18:15.026 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-06T00:18:15.204 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-06T00:18:15.228 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-06T00:18:15.270 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-06T00:18:15.589 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:18:15.597 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:18:15.745 INFO:teuthology.orchestra.run.smithi019.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-06T00:18:16.239 INFO:teuthology.orchestra.run.smithi019.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:18:17.159 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-06T00:18:17.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-06T00:18:17.165 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-06T00:18:30.469 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-06T00:18:30.588 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-06T00:18:31.357 INFO:teuthology.orchestra.run.smithi019.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-06T00:18:34.001 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-06T00:18:34.082 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-06T00:18:34.560 INFO:teuthology.orchestra.run.smithi019.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-06T00:18:36.169 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-06T00:18:36.329 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-06T00:18:36.932 INFO:teuthology.orchestra.run.smithi019.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-06T00:18:39.868 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-06T00:18:40.220 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-06T00:18:41.384 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-06T00:18:41.840 INFO:teuthology.orchestra.run.smithi019.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-06T00:18:44.777 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-06T00:18:44.798 INFO:teuthology.orchestra.run.smithi019.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-06T00:18:47.350 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-06T00:18:49.842 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-06T00:18:50.701 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-06T00:18:52.288 INFO:teuthology.orchestra.run.smithi019.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-06T00:18:52.918 INFO:teuthology.orchestra.run.smithi018.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-06T00:18:54.338 INFO:teuthology.orchestra.run.smithi019.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-06T00:18:54.343 INFO:tasks.kubeadm:Token: abcdef.ewc126tfy4amyzgm 2022-03-06T00:18:54.343 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): '172.21.15.18', Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): '172.21.15.19', Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): '172.21.15.59'} 2022-03-06T00:18:54.343 DEBUG:teuthology.orchestra.run.smithi018:> sudo kubeadm init --node-name smithi018 --token abcdef.ewc126tfy4amyzgm --pod-network-cidr 10.248.136.0/21 2022-03-06T00:18:54.584 INFO:teuthology.orchestra.run.smithi018.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-06T00:18:54.585 INFO:teuthology.orchestra.run.smithi018.stdout:[preflight] Running pre-flight checks 2022-03-06T00:18:54.978 INFO:teuthology.orchestra.run.smithi018.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-06T00:18:54.979 INFO:teuthology.orchestra.run.smithi018.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-06T00:18:54.979 INFO:teuthology.orchestra.run.smithi018.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-06T00:18:55.291 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-06T00:18:55.515 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "ca" certificate and key 2022-03-06T00:18:55.798 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "apiserver" certificate and key 2022-03-06T00:18:55.799 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi018] and IPs [10.96.0.1 172.21.15.18] 2022-03-06T00:18:55.908 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-06T00:18:56.086 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-06T00:18:56.681 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-06T00:18:57.125 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-06T00:18:57.279 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-06T00:18:57.279 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi018] and IPs [172.21.15.18 127.0.0.1 ::1] 2022-03-06T00:18:57.433 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-06T00:18:57.434 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi018] and IPs [172.21.15.18 127.0.0.1 ::1] 2022-03-06T00:18:58.000 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-06T00:18:58.262 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-06T00:18:58.394 INFO:teuthology.orchestra.run.smithi018.stdout:[certs] Generating "sa" key and public key 2022-03-06T00:18:58.395 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-06T00:18:59.028 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-06T00:18:59.377 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-06T00:18:59.668 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-06T00:18:59.867 INFO:teuthology.orchestra.run.smithi018.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-06T00:18:59.881 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-06T00:18:59.882 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-06T00:18:59.882 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet-start] Starting the kubelet 2022-03-06T00:19:00.110 INFO:teuthology.orchestra.run.smithi018.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-06T00:19:00.111 INFO:teuthology.orchestra.run.smithi018.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-06T00:19:00.114 INFO:teuthology.orchestra.run.smithi018.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-06T00:19:00.115 INFO:teuthology.orchestra.run.smithi018.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-06T00:19:00.116 INFO:teuthology.orchestra.run.smithi018.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-06T00:19:00.118 INFO:teuthology.orchestra.run.smithi018.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-06T00:19:17.121 INFO:teuthology.orchestra.run.smithi018.stdout:[apiclient] All control plane components are healthy after 17.003014 seconds 2022-03-06T00:19:17.122 INFO:teuthology.orchestra.run.smithi018.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-06T00:19:17.156 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-06T00:19:17.156 INFO:teuthology.orchestra.run.smithi018.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-06T00:19:17.697 INFO:teuthology.orchestra.run.smithi018.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-06T00:19:17.697 INFO:teuthology.orchestra.run.smithi018.stdout:[mark-control-plane] Marking the node smithi018 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-06T00:19:17.697 INFO:teuthology.orchestra.run.smithi018.stdout:[mark-control-plane] Marking the node smithi018 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-06T00:19:18.205 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] Using token: abcdef.ewc126tfy4amyzgm 2022-03-06T00:19:18.205 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-06T00:19:18.221 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-06T00:19:18.274 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-06T00:19:18.279 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-06T00:19:18.288 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-06T00:19:18.296 INFO:teuthology.orchestra.run.smithi018.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-06T00:19:18.407 INFO:teuthology.orchestra.run.smithi018.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-06T00:19:18.838 INFO:teuthology.orchestra.run.smithi018.stdout:[addons] Applied essential addon: CoreDNS 2022-03-06T00:19:19.005 INFO:teuthology.orchestra.run.smithi018.stdout:[addons] Applied essential addon: kube-proxy 2022-03-06T00:19:19.006 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:19:19.006 INFO:teuthology.orchestra.run.smithi018.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-06T00:19:19.007 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:19:19.007 INFO:teuthology.orchestra.run.smithi018.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-06T00:19:19.007 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:19:19.007 INFO:teuthology.orchestra.run.smithi018.stdout: mkdir -p $HOME/.kube 2022-03-06T00:19:19.008 INFO:teuthology.orchestra.run.smithi018.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-06T00:19:19.008 INFO:teuthology.orchestra.run.smithi018.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-06T00:19:19.008 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:19:19.008 INFO:teuthology.orchestra.run.smithi018.stdout:Alternatively, if you are the root user, you can run: 2022-03-06T00:19:19.009 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:19:19.009 INFO:teuthology.orchestra.run.smithi018.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-06T00:19:19.009 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:19:19.009 INFO:teuthology.orchestra.run.smithi018.stdout:You should now deploy a pod network to the cluster. 2022-03-06T00:19:19.009 INFO:teuthology.orchestra.run.smithi018.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-06T00:19:19.010 INFO:teuthology.orchestra.run.smithi018.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-06T00:19:19.010 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:19:19.010 INFO:teuthology.orchestra.run.smithi018.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-06T00:19:19.010 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:19:19.011 INFO:teuthology.orchestra.run.smithi018.stdout:kubeadm join 172.21.15.18:6443 --token abcdef.ewc126tfy4amyzgm \ 2022-03-06T00:19:19.011 INFO:teuthology.orchestra.run.smithi018.stdout: --discovery-token-ca-cert-hash sha256:7ae54d6bee977742a5229fdb47f549b686b6094f90a623354df7896a5d231b5c 2022-03-06T00:19:19.012 DEBUG:teuthology.orchestra.run.smithi019:> sudo kubeadm join 172.21.15.18:6443 --node-name smithi019 --token abcdef.ewc126tfy4amyzgm --discovery-token-unsafe-skip-ca-verification 2022-03-06T00:19:19.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo kubeadm join 172.21.15.18:6443 --node-name smithi059 --token abcdef.ewc126tfy4amyzgm --discovery-token-unsafe-skip-ca-verification 2022-03-06T00:19:19.055 INFO:teuthology.orchestra.run.smithi019.stdout:[preflight] Running pre-flight checks 2022-03-06T00:19:19.062 INFO:teuthology.orchestra.run.smithi059.stdout:[preflight] Running pre-flight checks 2022-03-06T00:19:31.886 INFO:teuthology.orchestra.run.smithi019.stdout:[preflight] Reading configuration from the cluster... 2022-03-06T00:19:31.887 INFO:teuthology.orchestra.run.smithi019.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-06T00:19:32.251 INFO:teuthology.orchestra.run.smithi019.stderr:W0306 00:19:32.251276 10654 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-06T00:19:32.269 INFO:teuthology.orchestra.run.smithi019.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-06T00:19:32.270 INFO:teuthology.orchestra.run.smithi019.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-06T00:19:32.270 INFO:teuthology.orchestra.run.smithi019.stdout:[kubelet-start] Starting the kubelet 2022-03-06T00:19:32.395 INFO:teuthology.orchestra.run.smithi059.stdout:[preflight] Reading configuration from the cluster... 2022-03-06T00:19:32.395 INFO:teuthology.orchestra.run.smithi059.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-06T00:19:32.506 INFO:teuthology.orchestra.run.smithi019.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-06T00:19:32.855 INFO:teuthology.orchestra.run.smithi059.stderr:W0306 00:19:32.855283 10643 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-06T00:19:33.171 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-06T00:19:33.172 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-06T00:19:33.172 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet-start] Starting the kubelet 2022-03-06T00:19:33.373 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-06T00:19:38.042 INFO:teuthology.orchestra.run.smithi019.stdout: 2022-03-06T00:19:38.042 INFO:teuthology.orchestra.run.smithi019.stdout:This node has joined the cluster: 2022-03-06T00:19:38.043 INFO:teuthology.orchestra.run.smithi019.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-06T00:19:38.043 INFO:teuthology.orchestra.run.smithi019.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-06T00:19:38.043 INFO:teuthology.orchestra.run.smithi019.stdout: 2022-03-06T00:19:38.043 INFO:teuthology.orchestra.run.smithi019.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-06T00:19:38.044 INFO:teuthology.orchestra.run.smithi019.stdout: 2022-03-06T00:19:38.902 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-03-06T00:19:38.902 INFO:teuthology.orchestra.run.smithi059.stdout:This node has joined the cluster: 2022-03-06T00:19:38.903 INFO:teuthology.orchestra.run.smithi059.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-06T00:19:38.903 INFO:teuthology.orchestra.run.smithi059.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-06T00:19:38.903 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-03-06T00:19:38.903 INFO:teuthology.orchestra.run.smithi059.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-06T00:19:38.904 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-03-06T00:19:38.908 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:38.909 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-06T00:19:38.924 INFO:tasks.kubeadm:Setting up kubectl 2022-03-06T00:19:38.924 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-06T00:19:38.989 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-06T00:19:39.007 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-06T00:19:39.021 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:39.021 DEBUG:teuthology.orchestra.run.smithi018:> dd of=.kube/config 2022-03-06T00:19:39.028 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:39.028 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/root/.kube/config 2022-03-06T00:19:39.088 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:19:39.088 DEBUG:teuthology.orchestra.run.smithi019:> dd of=.kube/config 2022-03-06T00:19:39.096 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:19:39.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/root/.kube/config 2022-03-06T00:19:39.154 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:19:39.155 DEBUG:teuthology.orchestra.run.smithi059:> dd of=.kube/config 2022-03-06T00:19:39.162 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:19:39.162 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/root/.kube/config 2022-03-06T00:19:39.175 DEBUG:teuthology.orchestra.run.smithi018:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-06T00:19:41.682 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-06T00:19:41.767 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-06T00:19:41.813 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-06T00:19:41.927 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-06T00:19:41.934 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-06T00:19:41.991 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-06T00:19:42.116 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-06T00:19:42.121 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-06T00:19:42.173 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-06T00:19:42.290 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-06T00:19:42.339 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-06T00:19:42.435 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-06T00:19:42.463 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-06T00:19:42.515 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-06T00:19:42.609 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-06T00:19:42.663 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-06T00:19:42.764 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-06T00:19:42.815 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-06T00:19:42.910 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-06T00:19:42.975 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-06T00:19:43.076 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-06T00:19:43.117 INFO:teuthology.orchestra.run.smithi018.stdout:namespace/tigera-operator created 2022-03-06T00:19:43.126 INFO:teuthology.orchestra.run.smithi018.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-06T00:19:43.135 INFO:teuthology.orchestra.run.smithi018.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-06T00:19:43.162 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/tigera-operator created 2022-03-06T00:19:43.260 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-06T00:19:43.293 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-06T00:19:43.308 INFO:teuthology.orchestra.run.smithi018.stdout:deployment.apps/tigera-operator created 2022-03-06T00:19:43.314 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f - 2022-03-06T00:19:45.967 INFO:teuthology.orchestra.run.smithi018.stdout:installation.operator.tigera.io/default created 2022-03-06T00:19:45.973 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-06T00:19:45.974 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:45.974 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:45.983 INFO:tasks.kubeadm:Scratch devices on smithi018: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:19:45.984 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-06T00:19:46.012 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records in 2022-03-06T00:19:46.012 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records out 2022-03-06T00:19:46.012 INFO:teuthology.orchestra.run.smithi018.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0142392 s, 736 MB/s 2022-03-06T00:19:46.013 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-06T00:19:46.084 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records in 2022-03-06T00:19:46.085 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records out 2022-03-06T00:19:46.085 INFO:teuthology.orchestra.run.smithi018.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0107275 s, 977 MB/s 2022-03-06T00:19:46.086 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-06T00:19:46.112 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records in 2022-03-06T00:19:46.112 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records out 2022-03-06T00:19:46.112 INFO:teuthology.orchestra.run.smithi018.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0135793 s, 772 MB/s 2022-03-06T00:19:46.114 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-06T00:19:46.137 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records in 2022-03-06T00:19:46.137 INFO:teuthology.orchestra.run.smithi018.stderr:10+0 records out 2022-03-06T00:19:46.137 INFO:teuthology.orchestra.run.smithi018.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124999 s, 839 MB/s 2022-03-06T00:19:46.139 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:19:46.139 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:46.147 INFO:tasks.kubeadm:Scratch devices on smithi019: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:19:46.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-06T00:19:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records in 2022-03-06T00:19:46.178 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2022-03-06T00:19:46.179 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0166572 s, 630 MB/s 2022-03-06T00:19:46.180 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-06T00:19:46.253 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records in 2022-03-06T00:19:46.253 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2022-03-06T00:19:46.254 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.016279 s, 644 MB/s 2022-03-06T00:19:46.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-06T00:19:46.287 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records in 2022-03-06T00:19:46.287 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2022-03-06T00:19:46.287 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0178677 s, 587 MB/s 2022-03-06T00:19:46.289 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-06T00:19:46.318 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records in 2022-03-06T00:19:46.318 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2022-03-06T00:19:46.318 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0152178 s, 689 MB/s 2022-03-06T00:19:46.320 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:19:46.320 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:46.328 INFO:tasks.kubeadm:Scratch devices on smithi059: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:19:46.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-06T00:19:46.395 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records in 2022-03-06T00:19:46.395 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-03-06T00:19:46.395 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123065 s, 852 MB/s 2022-03-06T00:19:46.397 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-06T00:19:46.422 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records in 2022-03-06T00:19:46.423 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-03-06T00:19:46.423 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155472 s, 674 MB/s 2022-03-06T00:19:46.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-06T00:19:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records in 2022-03-06T00:19:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-03-06T00:19:46.453 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0169198 s, 620 MB/s 2022-03-06T00:19:46.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-06T00:19:46.487 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records in 2022-03-06T00:19:46.488 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-03-06T00:19:46.488 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0179255 s, 585 MB/s 2022-03-06T00:19:46.527 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-06T00:19:46.528 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi018-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi018 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi018-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi018 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi018-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi018 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi018-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi018 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi019-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi019 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi019-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi019 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi019-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi019 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi019-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi019 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi059-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi059 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi059-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi059 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi059-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi059 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi059-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi059 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-06T00:19:46.528 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f - 2022-03-06T00:19:47.403 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-1 created 2022-03-06T00:19:47.424 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-2 created 2022-03-06T00:19:47.434 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-3 created 2022-03-06T00:19:47.453 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-4 created 2022-03-06T00:19:47.492 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi019-lv-1 created 2022-03-06T00:19:47.498 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi019-lv-2 created 2022-03-06T00:19:47.509 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi019-lv-3 created 2022-03-06T00:19:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi019-lv-4 created 2022-03-06T00:19:47.540 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi059-lv-1 created 2022-03-06T00:19:47.551 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi059-lv-2 created 2022-03-06T00:19:47.574 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi059-lv-3 created 2022-03-06T00:19:47.585 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi059-lv-4 created 2022-03-06T00:19:47.633 INFO:teuthology.orchestra.run.smithi018.stdout:storageclass.storage.k8s.io/scratch created 2022-03-06T00:19:47.639 DEBUG:teuthology.orchestra.run.smithi018:> kubectl taint node smithi018 node-role.kubernetes.io/master- || true 2022-03-06T00:19:47.709 INFO:teuthology.orchestra.run.smithi018.stdout:node/smithi018 untainted 2022-03-06T00:19:47.711 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-06T00:19:47.712 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-06T00:19:47.725 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-06T00:19:47.726 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:47.726 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:47.734 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:19:47.734 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-06T00:19:47.740 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-06T00:19:47.740 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:47.740 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-06T00:19:47.741 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:47.741 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2022-03-06 00:19:46.012046491 +0000 2022-03-06T00:19:47.741 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2022-03-06 00:19:46.012046491 +0000 2022-03-06T00:19:47.742 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2022-03-06 00:19:46.012046491 +0000 2022-03-06T00:19:47.742 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2022-03-06T00:19:47.748 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2022-03-06T00:19:47.748 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2022-03-06T00:19:47.749 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.00023231 s, 2.2 MB/s 2022-03-06T00:19:47.753 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-06T00:19:47.806 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-06T00:19:47.807 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:47.807 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-03-06T00:19:47.807 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:47.807 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2022-03-06 00:19:46.088046490 +0000 2022-03-06T00:19:47.808 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2022-03-06 00:19:46.088046490 +0000 2022-03-06T00:19:47.808 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2022-03-06 00:19:46.088046490 +0000 2022-03-06T00:19:47.808 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2022-03-06T00:19:47.815 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2022-03-06T00:19:47.816 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2022-03-06T00:19:47.816 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000229458 s, 2.2 MB/s 2022-03-06T00:19:47.820 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-06T00:19:47.826 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-06T00:19:47.826 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:47.826 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-03-06T00:19:47.827 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:47.827 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2022-03-06 00:19:46.112046490 +0000 2022-03-06T00:19:47.827 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2022-03-06 00:19:46.112046490 +0000 2022-03-06T00:19:47.827 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2022-03-06 00:19:46.112046490 +0000 2022-03-06T00:19:47.828 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2022-03-06T00:19:47.833 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2022-03-06T00:19:47.833 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2022-03-06T00:19:47.834 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000265103 s, 1.9 MB/s 2022-03-06T00:19:47.838 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-06T00:19:47.845 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-06T00:19:47.845 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:47.845 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-06T00:19:47.845 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:47.846 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2022-03-06 00:19:46.140046489 +0000 2022-03-06T00:19:47.846 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2022-03-06 00:19:46.140046489 +0000 2022-03-06T00:19:47.846 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2022-03-06 00:19:46.140046489 +0000 2022-03-06T00:19:47.846 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2022-03-06T00:19:47.852 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2022-03-06T00:19:47.853 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2022-03-06T00:19:47.853 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.00033129 s, 1.5 MB/s 2022-03-06T00:19:47.857 DEBUG:teuthology.orchestra.run.smithi018:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-06T00:19:47.954 INFO:teuthology.orchestra.run.smithi018.stdout:loop 2022-03-06T00:19:47.955 INFO:tasks.nvme_loop:Connecting nvme_loop smithi018:/dev/vg_nvme/lv_1... 2022-03-06T00:19:47.956 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-06T00:19:47.976 INFO:teuthology.orchestra.run.smithi018.stdout:1 2022-03-06T00:19:48.003 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_11 2022-03-06T00:19:48.026 INFO:tasks.nvme_loop:Connecting nvme_loop smithi018:/dev/vg_nvme/lv_2... 2022-03-06T00:19:48.026 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-06T00:19:48.089 INFO:teuthology.orchestra.run.smithi018.stdout:1 2022-03-06T00:19:48.113 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_21 2022-03-06T00:19:48.135 INFO:tasks.nvme_loop:Connecting nvme_loop smithi018:/dev/vg_nvme/lv_3... 2022-03-06T00:19:48.136 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-06T00:19:48.157 INFO:teuthology.orchestra.run.smithi018.stdout:1 2022-03-06T00:19:48.180 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_31 2022-03-06T00:19:48.204 INFO:tasks.nvme_loop:Connecting nvme_loop smithi018:/dev/vg_nvme/lv_4... 2022-03-06T00:19:48.204 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-06T00:19:48.225 INFO:teuthology.orchestra.run.smithi018.stdout:1 2022-03-06T00:19:48.250 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_41 2022-03-06T00:19:48.274 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:48.275 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:48.322 DEBUG:teuthology.orchestra.run.smithi018:> sudo nvme list 2022-03-06T00:19:48.335 INFO:teuthology.orchestra.run.smithi018.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-06T00:19:48.336 INFO:teuthology.orchestra.run.smithi018.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-06T00:19:48.336 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme0n1 PHMB75120091480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-06T00:19:48.336 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme1n1 fb9c267b3ea7e1b3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:48.336 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme2n1 27901a95cf16fba0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:48.337 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme3n1 366e821eaa07c0f0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:48.337 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/nvme4n1 a26dabb887a0129d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:48.338 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-06T00:19:48.338 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:48.339 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/scratch_devs 2022-03-06T00:19:48.351 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:19:48.351 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:48.359 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:19:48.359 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-06T00:19:48.367 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-06T00:19:48.367 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:48.368 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-06T00:19:48.368 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:48.368 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2022-03-06 00:19:46.182366180 +0000 2022-03-06T00:19:48.368 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2022-03-06 00:19:46.182366180 +0000 2022-03-06T00:19:48.369 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2022-03-06 00:19:46.182366180 +0000 2022-03-06T00:19:48.369 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2022-03-06T00:19:48.379 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2022-03-06T00:19:48.379 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2022-03-06T00:19:48.380 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000601494 s, 851 kB/s 2022-03-06T00:19:48.384 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-06T00:19:48.437 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-06T00:19:48.438 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:48.438 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-06T00:19:48.438 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:48.438 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2022-03-06 00:19:46.258366178 +0000 2022-03-06T00:19:48.439 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2022-03-06 00:19:46.258366178 +0000 2022-03-06T00:19:48.439 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2022-03-06 00:19:46.258366178 +0000 2022-03-06T00:19:48.439 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2022-03-06T00:19:48.447 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2022-03-06T00:19:48.447 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2022-03-06T00:19:48.448 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000494941 s, 1.0 MB/s 2022-03-06T00:19:48.452 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-06T00:19:48.458 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-06T00:19:48.459 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:48.459 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-06T00:19:48.459 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:48.459 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2022-03-06 00:19:46.290366177 +0000 2022-03-06T00:19:48.460 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2022-03-06 00:19:46.290366177 +0000 2022-03-06T00:19:48.460 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2022-03-06 00:19:46.290366177 +0000 2022-03-06T00:19:48.463 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2022-03-06T00:19:48.470 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2022-03-06T00:19:48.470 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2022-03-06T00:19:48.470 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000492178 s, 1.0 MB/s 2022-03-06T00:19:48.475 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-06T00:19:48.481 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-06T00:19:48.482 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:48.482 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-06T00:19:48.482 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:48.483 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2022-03-06 00:19:46.322366176 +0000 2022-03-06T00:19:48.483 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2022-03-06 00:19:46.322366176 +0000 2022-03-06T00:19:48.483 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2022-03-06 00:19:46.322366176 +0000 2022-03-06T00:19:48.483 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2022-03-06T00:19:48.491 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2022-03-06T00:19:48.492 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2022-03-06T00:19:48.492 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.00055931 s, 915 kB/s 2022-03-06T00:19:48.496 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-06T00:19:48.571 INFO:teuthology.orchestra.run.smithi019.stdout:loop 2022-03-06T00:19:48.572 INFO:tasks.nvme_loop:Connecting nvme_loop smithi019:/dev/vg_nvme/lv_1... 2022-03-06T00:19:48.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-06T00:19:48.599 INFO:teuthology.orchestra.run.smithi019.stdout:1 2022-03-06T00:19:48.626 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/vg_nvme/lv_11 2022-03-06T00:19:48.649 INFO:tasks.nvme_loop:Connecting nvme_loop smithi019:/dev/vg_nvme/lv_2... 2022-03-06T00:19:48.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-06T00:19:48.675 INFO:teuthology.orchestra.run.smithi019.stdout:1 2022-03-06T00:19:48.699 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/vg_nvme/lv_21 2022-03-06T00:19:48.722 INFO:tasks.nvme_loop:Connecting nvme_loop smithi019:/dev/vg_nvme/lv_3... 2022-03-06T00:19:48.723 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-06T00:19:48.789 INFO:teuthology.orchestra.run.smithi019.stdout:1 2022-03-06T00:19:48.816 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/vg_nvme/lv_31 2022-03-06T00:19:48.841 INFO:tasks.nvme_loop:Connecting nvme_loop smithi019:/dev/vg_nvme/lv_4... 2022-03-06T00:19:48.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-06T00:19:48.862 INFO:teuthology.orchestra.run.smithi019.stdout:1 2022-03-06T00:19:48.888 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/vg_nvme/lv_41 2022-03-06T00:19:48.909 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:19:48.909 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:48.916 DEBUG:teuthology.orchestra.run.smithi019:> sudo nvme list 2022-03-06T00:19:48.927 INFO:teuthology.orchestra.run.smithi019.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-06T00:19:48.928 INFO:teuthology.orchestra.run.smithi019.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-06T00:19:48.928 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme0n1 CVFT5330002G400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-06T00:19:48.928 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme1n1 c8bfcd4e5675db79 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:48.928 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme2n1 59bdebcf65147e8c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:48.929 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme3n1 766768ff60b65632 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:48.930 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-06T00:19:49.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo nvme list 2022-03-06T00:19:49.946 INFO:teuthology.orchestra.run.smithi019.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-06T00:19:49.947 INFO:teuthology.orchestra.run.smithi019.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-06T00:19:49.947 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme0n1 CVFT5330002G400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-06T00:19:49.947 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme1n1 c8bfcd4e5675db79 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:49.947 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme2n1 59bdebcf65147e8c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:49.948 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme3n1 766768ff60b65632 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:49.948 INFO:teuthology.orchestra.run.smithi019.stdout:/dev/nvme4n1 7df018d7b646c9da Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:49.949 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-06T00:19:49.949 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:19:49.950 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/scratch_devs 2022-03-06T00:19:50.007 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:19:50.007 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:50.015 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:19:50.015 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-06T00:19:50.021 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-06T00:19:50.022 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:50.022 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-06T00:19:50.022 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:50.022 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-03-06 00:19:46.395517218 +0000 2022-03-06T00:19:50.023 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-03-06 00:19:46.395517218 +0000 2022-03-06T00:19:50.023 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-03-06 00:19:46.395517218 +0000 2022-03-06T00:19:50.023 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-03-06T00:19:50.032 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-03-06T00:19:50.032 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-03-06T00:19:50.032 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000649128 s, 789 kB/s 2022-03-06T00:19:50.036 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-06T00:19:50.042 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-06T00:19:50.042 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:50.042 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-06T00:19:50.043 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:50.043 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-03-06 00:19:46.423517217 +0000 2022-03-06T00:19:50.043 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-03-06 00:19:46.423517217 +0000 2022-03-06T00:19:50.043 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-03-06 00:19:46.423517217 +0000 2022-03-06T00:19:50.044 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-03-06T00:19:50.050 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-03-06T00:19:50.050 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-03-06T00:19:50.050 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000485453 s, 1.1 MB/s 2022-03-06T00:19:50.054 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-06T00:19:50.106 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-06T00:19:50.106 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:50.106 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-03-06T00:19:50.106 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:50.107 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-03-06 00:19:46.455517216 +0000 2022-03-06T00:19:50.107 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-03-06 00:19:46.455517216 +0000 2022-03-06T00:19:50.107 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-03-06 00:19:46.455517216 +0000 2022-03-06T00:19:50.107 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-03-06T00:19:50.116 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-03-06T00:19:50.116 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-03-06T00:19:50.116 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000495174 s, 1.0 MB/s 2022-03-06T00:19:50.120 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-06T00:19:50.127 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-06T00:19:50.127 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:19:50.127 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 517 Links: 1 2022-03-06T00:19:50.127 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:19:50.128 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-03-06 00:19:46.491517215 +0000 2022-03-06T00:19:50.128 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-03-06 00:19:46.487517216 +0000 2022-03-06T00:19:50.128 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-03-06 00:19:46.487517216 +0000 2022-03-06T00:19:50.128 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-03-06T00:19:50.135 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-03-06T00:19:50.135 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-03-06T00:19:50.135 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000547196 s, 936 kB/s 2022-03-06T00:19:50.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-06T00:19:50.227 INFO:teuthology.orchestra.run.smithi059.stdout:loop 2022-03-06T00:19:50.229 INFO:tasks.nvme_loop:Connecting nvme_loop smithi059:/dev/vg_nvme/lv_1... 2022-03-06T00:19:50.229 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-06T00:19:50.249 INFO:teuthology.orchestra.run.smithi059.stdout:1 2022-03-06T00:19:50.275 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/vg_nvme/lv_11 2022-03-06T00:19:50.297 INFO:tasks.nvme_loop:Connecting nvme_loop smithi059:/dev/vg_nvme/lv_2... 2022-03-06T00:19:50.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-06T00:19:50.317 INFO:teuthology.orchestra.run.smithi059.stdout:1 2022-03-06T00:19:50.345 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/vg_nvme/lv_21 2022-03-06T00:19:50.368 INFO:tasks.nvme_loop:Connecting nvme_loop smithi059:/dev/vg_nvme/lv_3... 2022-03-06T00:19:50.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-06T00:19:50.387 INFO:teuthology.orchestra.run.smithi059.stdout:1 2022-03-06T00:19:50.411 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/vg_nvme/lv_31 2022-03-06T00:19:50.432 INFO:tasks.nvme_loop:Connecting nvme_loop smithi059:/dev/vg_nvme/lv_4... 2022-03-06T00:19:50.432 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-06T00:19:50.453 INFO:teuthology.orchestra.run.smithi059.stdout:1 2022-03-06T00:19:50.479 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/vg_nvme/lv_41 2022-03-06T00:19:50.502 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:19:50.503 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:50.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo nvme list 2022-03-06T00:19:50.567 INFO:teuthology.orchestra.run.smithi059.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-06T00:19:50.567 INFO:teuthology.orchestra.run.smithi059.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-06T00:19:50.568 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme0n1 CVFT5330009L400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-06T00:19:50.568 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme1n1 1a216d2c14364059 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:50.568 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme2n1 c158461fbc02e859 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:50.568 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme3n1 9fb1e398169ef0a5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:50.569 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme4n1 542aeed5df211a7c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:19:50.570 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-06T00:19:50.570 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:19:50.571 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/scratch_devs 2022-03-06T00:19:50.584 INFO:teuthology.run_tasks:Running task rook... 2022-03-06T00:19:50.668 INFO:tasks.rook:Rook start 2022-03-06T00:19:50.669 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'd0dc2fbbf3f64a2d513e2355b245d0ea9728ff97', 'spec': {'mon': {'allowMultiplePerNode': False}, 'network': {'provider': 'host'}}, 'rook_image': 'rook/ceph:master'} 2022-03-06T00:19:50.669 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 2022-03-06T00:19:50.669 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-06T00:19:50.670 DEBUG:teuthology.orchestra.run.smithi018:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-06T00:19:50.679 INFO:teuthology.orchestra.run.smithi018.stderr:Cloning into 'rook'... 2022-03-06T00:19:53.103 INFO:tasks.rook:/home/teuthworker 2022-03-06T00:19:53.104 DEBUG:teuthology.orchestra.run.smithi018:> PATH=/sbin:/usr/sbin ip addr show 2022-03-06T00:19:53.151 INFO:teuthology.orchestra.run.smithi018.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-06T00:19:53.151 INFO:teuthology.orchestra.run.smithi018.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-06T00:19:53.151 INFO:teuthology.orchestra.run.smithi018.stdout: inet 127.0.0.1/8 scope host lo 2022-03-06T00:19:53.152 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft forever preferred_lft forever 2022-03-06T00:19:53.152 INFO:teuthology.orchestra.run.smithi018.stdout: inet6 ::1/128 scope host 2022-03-06T00:19:53.152 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft forever preferred_lft forever 2022-03-06T00:19:53.152 INFO:teuthology.orchestra.run.smithi018.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-06T00:19:53.153 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 0c:c4:7a:6b:fd:98 brd ff:ff:ff:ff:ff:ff 2022-03-06T00:19:53.153 INFO:teuthology.orchestra.run.smithi018.stdout: altname enp7s0f0 2022-03-06T00:19:53.153 INFO:teuthology.orchestra.run.smithi018.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-06T00:19:53.153 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 0c:c4:7a:6b:fd:99 brd ff:ff:ff:ff:ff:ff 2022-03-06T00:19:53.153 INFO:teuthology.orchestra.run.smithi018.stdout: altname enp7s0f1 2022-03-06T00:19:53.154 INFO:teuthology.orchestra.run.smithi018.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-06T00:19:53.154 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 0c:c4:7a:bd:13:62 brd ff:ff:ff:ff:ff:ff 2022-03-06T00:19:53.154 INFO:teuthology.orchestra.run.smithi018.stdout: altname enp1s0f0 2022-03-06T00:19:53.154 INFO:teuthology.orchestra.run.smithi018.stdout: inet 172.21.15.18/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-06T00:19:53.155 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft 42857sec preferred_lft 42857sec 2022-03-06T00:19:53.155 INFO:teuthology.orchestra.run.smithi018.stdout: inet6 fe80::ec4:7aff:febd:1362/64 scope link 2022-03-06T00:19:53.155 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft forever preferred_lft forever 2022-03-06T00:19:53.156 INFO:teuthology.orchestra.run.smithi018.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-06T00:19:53.156 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 0c:c4:7a:bd:13:63 brd ff:ff:ff:ff:ff:ff 2022-03-06T00:19:53.157 INFO:teuthology.orchestra.run.smithi018.stdout: altname enp1s0f1 2022-03-06T00:19:53.157 INFO:teuthology.orchestra.run.smithi018.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-06T00:19:53.157 INFO:teuthology.orchestra.run.smithi018.stdout: link/ether 02:42:09:be:fd:e5 brd ff:ff:ff:ff:ff:ff 2022-03-06T00:19:53.157 INFO:teuthology.orchestra.run.smithi018.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-06T00:19:53.158 INFO:teuthology.orchestra.run.smithi018.stdout: valid_lft forever preferred_lft forever 2022-03-06T00:19:53.159 DEBUG:teuthology.orchestra.run.smithi018:> which systemctl 2022-03-06T00:19:53.164 INFO:teuthology.orchestra.run.smithi018.stdout:/usr/bin/systemctl 2022-03-06T00:19:53.164 DEBUG:teuthology.orchestra.run.smithi018:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-06T00:19:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-06T00:19:53.169 DEBUG:teuthology.orchestra.run.smithi018:> true 2022-03-06T00:19:53.247 DEBUG:teuthology.orchestra.run.smithi018:> sudo lsb_release -is 2022-03-06T00:19:53.297 INFO:teuthology.orchestra.run.smithi018.stdout:Ubuntu 2022-03-06T00:19:53.302 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-06T00:19:53.303 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-06T00:19:53.315 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:19:53.329 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:53.330 DEBUG:teuthology.orchestra.run.smithi018:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-06T00:19:53.339 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-06T00:19:53.384 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:19:53.385 DEBUG:teuthology.orchestra.run.smithi018:> dd of=operator.yaml 2022-03-06T00:19:53.393 INFO:tasks.rook:Deploying operator 2022-03-06T00:19:53.404 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:19:53.432 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:19:53.491 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-06T00:19:53.752 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-06T00:19:53.779 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-06T00:19:53.805 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-06T00:19:53.887 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-06T00:19:53.994 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-06T00:19:54.060 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-06T00:19:54.125 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-06T00:19:54.144 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-06T00:19:54.233 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-06T00:19:54.262 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-06T00:19:54.320 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-06T00:19:54.397 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-06T00:19:54.401 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-06T00:19:54.433 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-06T00:19:54.464 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-06T00:19:54.481 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-06T00:19:54.563 INFO:teuthology.orchestra.run.smithi018.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-06T00:19:54.588 INFO:teuthology.orchestra.run.smithi018.stdout:namespace/rook-ceph created 2022-03-06T00:19:54.605 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-06T00:19:54.621 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-06T00:19:54.637 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-06T00:19:54.733 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-06T00:19:54.756 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-06T00:19:54.771 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-06T00:19:54.783 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-06T00:19:54.805 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-06T00:19:54.816 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-06T00:19:54.824 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-06T00:19:54.874 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-06T00:19:54.878 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-06T00:19:54.882 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-06T00:19:54.905 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-06T00:19:54.916 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-06T00:19:54.924 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-06T00:19:54.933 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-06T00:19:54.941 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-06T00:19:54.949 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-06T00:19:54.972 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-06T00:19:55.007 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-06T00:19:55.010 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-06T00:19:55.016 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-06T00:19:55.024 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-06T00:19:55.047 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-06T00:19:55.057 INFO:teuthology.orchestra.run.smithi018.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-06T00:19:55.066 INFO:teuthology.orchestra.run.smithi018.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-06T00:19:55.067 INFO:teuthology.orchestra.run.smithi018.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-06T00:19:55.074 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-06T00:19:55.082 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-06T00:19:55.091 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-06T00:19:55.147 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-06T00:19:55.148 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-06T00:19:55.150 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-06T00:19:55.157 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-06T00:19:55.166 INFO:teuthology.orchestra.run.smithi018.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-06T00:19:55.174 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-06T00:19:55.199 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-06T00:19:55.208 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-06T00:19:55.216 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-06T00:19:55.224 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-06T00:19:55.233 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-06T00:19:55.241 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-06T00:19:55.299 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-06T00:19:55.302 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-06T00:19:55.308 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-06T00:19:55.316 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-06T00:19:55.324 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-06T00:19:55.332 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-06T00:19:55.341 INFO:teuthology.orchestra.run.smithi018.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-06T00:19:55.363 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-06T00:19:55.375 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-mgr created 2022-03-06T00:19:55.382 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-osd created 2022-03-06T00:19:55.391 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-06T00:19:55.450 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-ceph-system created 2022-03-06T00:19:55.455 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-06T00:19:55.466 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-06T00:19:55.489 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-06T00:19:55.515 INFO:teuthology.orchestra.run.smithi018.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-06T00:19:55.539 INFO:teuthology.orchestra.run.smithi018.stdout:configmap/rook-ceph-operator-config created 2022-03-06T00:19:55.605 INFO:teuthology.orchestra.run.smithi018.stdout:deployment.apps/rook-ceph-operator created 2022-03-06T00:19:55.610 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:19:55.695 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:55.695 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-d4cp4 0/1 Pending 0 0s 2022-03-06T00:20:05.699 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:20:06.046 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:06.047 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-d4cp4 0/1 Pending 0 11s 2022-03-06T00:20:16.049 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:20:16.138 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:16.139 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-d4cp4 0/1 Pending 0 21s 2022-03-06T00:20:26.143 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:20:26.360 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:26.361 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-d4cp4 0/1 ContainerCreating 0 31s 2022-03-06T00:20:36.364 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:20:36.485 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:36.485 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-d4cp4 0/1 ContainerCreating 0 41s 2022-03-06T00:20:46.492 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:20:46.566 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:46.566 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-d4cp4 0/1 ContainerCreating 0 51s 2022-03-06T00:20:56.569 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:20:56.634 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:56.635 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-d4cp4 0/1 ContainerCreating 0 61s 2022-03-06T00:21:06.638 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:21:06.698 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:06.699 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-operator-75dd789779-d4cp4 1/1 Running 0 71s 2022-03-06T00:21:06.701 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-d4cp4 2022-03-06T00:21:06.710 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:21:06.711 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:21:06.765 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:21:06.766 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:21:06.774 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:21:06.775 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:21:06.784 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-06T00:21:06.784 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-06T00:21:06.785 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-06T00:21:06.785 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-06T00:21:06.785 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-06T00:21:06.786 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-06T00:21:06.786 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-06T00:21:06.787 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-06T00:21:06.789 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f - 2022-03-06T00:21:06.792 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.114428 I | rookcmd: starting Rook v1.8.0-alpha.0.373.g507a2e336 with arguments '/usr/local/bin/rook ceph operator' 2022-03-06T00:21:06.793 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.114556 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-06T00:21:06.793 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.114559 I | cephcmd: starting Rook-Ceph operator 2022-03-06T00:21:06.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.248619 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-06T00:21:06.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.257233 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:21:06.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.257262 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:21:06.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.257360 I | operator: setting up schemes 2022-03-06T00:21:06.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.260417 I | operator: setting up the controller-runtime manager 2022-03-06T00:21:06.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.567136 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:21:06.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.569808 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:21:06.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.569852 I | ceph-cluster-controller: successfully started 2022-03-06T00:21:06.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.569925 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:21:06.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.569954 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:21:06.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.569977 I | ceph-block-pool-controller: successfully started 2022-03-06T00:21:06.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570007 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:21:06.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570028 I | ceph-object-realm-controller: successfully started 2022-03-06T00:21:06.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570048 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:21:06.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570063 I | ceph-object-zone-controller: successfully started 2022-03-06T00:21:06.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570173 I | ceph-object-controller: successfully started 2022-03-06T00:21:06.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570219 I | ceph-file-controller: successfully started 2022-03-06T00:21:06.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570276 I | ceph-nfs-controller: successfully started 2022-03-06T00:21:06.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570309 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:21:06.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570342 I | ceph-client-controller: successfully started 2022-03-06T00:21:06.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570370 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:21:06.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570413 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:21:06.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570435 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:21:06.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570448 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:21:06.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570474 I | ceph-bucket-topic: successfully started 2022-03-06T00:21:06.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570490 I | ceph-bucket-notification: successfully started 2022-03-06T00:21:06.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570500 I | ceph-bucket-notification: successfully started 2022-03-06T00:21:06.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.570512 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:21:06.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.571438 I | operator: starting the controller-runtime manager 2022-03-06T00:21:06.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.675736 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:21:06.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.675771 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:21:06.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.675784 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:21:06.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.675962 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-06T00:21:06.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.675986 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-06T00:21:06.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.696827 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:21:06.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.696856 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-06T00:21:06.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.702600 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-06T00:21:06.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.702619 I | ceph-csi: CSI CephFS driver disabled 2022-03-06T00:21:06.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.702628 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-06T00:21:06.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.704554 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-06T00:21:06.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:06.741169 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-06T00:21:07.819 INFO:teuthology.orchestra.run.smithi018.stdout:configmap/rook-config-override created 2022-03-06T00:21:07.823 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: false count: 3 network: provider: host skipUpgradeChecks: true 2022-03-06T00:21:07.824 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:21:07.824 DEBUG:teuthology.orchestra.run.smithi018:> dd of=cluster.yaml 2022-03-06T00:21:07.830 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f cluster.yaml 2022-03-06T00:21:10.056 INFO:teuthology.orchestra.run.smithi018.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-06T00:21:10.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.063016 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-06T00:21:10.069 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:21:10.081 DEBUG:teuthology.orchestra.run.smithi018:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-06T00:21:10.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.096127 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:21:10.113 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.113229 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-06T00:21:10.125 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.126535 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-06T00:21:10.138 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.139656 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:21:10.309 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309583 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-06T00:21:10.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309627 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-06T00:21:10.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309637 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-06T00:21:10.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309645 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-06T00:21:10.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309655 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-06T00:21:10.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309679 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-06T00:21:10.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309702 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-06T00:21:10.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309721 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-06T00:21:10.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309735 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-06T00:21:10.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309753 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-06T00:21:10.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309770 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-06T00:21:10.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309785 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-06T00:21:10.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309881 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-06T00:21:10.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309965 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-06T00:21:10.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.309988 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-06T00:21:10.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.310004 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-06T00:21:10.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.310107 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:21:10.314 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:21:10.310129 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:21:10.357 INFO:teuthology.orchestra.run.smithi018.stdout:deployment.apps/rook-ceph-tools created 2022-03-06T00:21:10.362 INFO:tasks.rook:Waiting for tools container to start 2022-03-06T00:21:10.363 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:10.485 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:10.486 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 Pending 0 0s 2022-03-06T00:21:15.497 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:15.567 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:15.567 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 5s 2022-03-06T00:21:20.571 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:20.640 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:20.641 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 10s 2022-03-06T00:21:25.643 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:25.709 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:25.709 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 15s 2022-03-06T00:21:30.712 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:30.775 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:30.776 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 20s 2022-03-06T00:21:35.778 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:35.848 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:35.848 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 25s 2022-03-06T00:21:40.851 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:40.922 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:40.923 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 30s 2022-03-06T00:21:45.925 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:45.990 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:45.991 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 35s 2022-03-06T00:21:50.994 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:51.062 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:51.063 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 41s 2022-03-06T00:21:56.065 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:56.131 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:56.132 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 46s 2022-03-06T00:22:01.134 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:01.200 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:01.200 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 51s 2022-03-06T00:22:06.203 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:06.267 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:06.268 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 56s 2022-03-06T00:22:11.270 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:11.343 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:11.343 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 61s 2022-03-06T00:22:16.347 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:16.412 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:16.413 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 66s 2022-03-06T00:22:21.416 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:21.488 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:21.488 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 71s 2022-03-06T00:22:26.491 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:26.560 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:26.560 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 76s 2022-03-06T00:22:31.563 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:31.637 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:31.637 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 81s 2022-03-06T00:22:36.639 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:36.709 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:36.710 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 86s 2022-03-06T00:22:41.712 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:41.779 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:41.779 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 91s 2022-03-06T00:22:46.783 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:46.849 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:46.849 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 96s 2022-03-06T00:22:51.231 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239188 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-06T00:22:51.232 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239218 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-06T00:22:51.232 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239225 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:22:51.232 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239229 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:22:51.232 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239233 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:22:51.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239236 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:22:51.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239239 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:22:51.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239242 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:22:51.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239246 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:22:51.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239252 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:22:51.234 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239257 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:22:51.234 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239262 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:22:51.234 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239268 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:22:51.234 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239273 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:22:51.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239279 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-06T00:22:51.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239285 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-06T00:22:51.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239290 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-06T00:22:51.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239295 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-06T00:22:51.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239301 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-06T00:22:51.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239309 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:22:51.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239315 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:22:51.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239321 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-06T00:22:51.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239326 I | ceph-csi: Kubernetes version is 1.23 2022-03-06T00:22:51.238 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239334 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-06T00:22:51.238 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.239344 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-06T00:22:51.261 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.270453 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-06T00:22:51.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.276323 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:22:51.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.276344 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:22:51.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.276350 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:22:51.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.276354 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:22:51.269 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.276360 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:22:51.269 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.276365 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:22:51.269 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.276371 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-06T00:22:51.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.306537 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:22:51.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.306563 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:22:51.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.306569 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:22:51.421 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.430004 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-06T00:22:51.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.568778 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:22:51.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.568876 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:22:51.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.568924 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-06T00:22:51.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.648380 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:22:51.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.648406 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:22:51.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.648413 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:22:51.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:51.768999 I | ceph-csi: successfully started CSI CephFS driver 2022-03-06T00:22:51.852 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:52.252 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:52.252 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 102s 2022-03-06T00:22:52.670 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:52.678955 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:22:53.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:53.050635 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-06T00:22:53.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:53.050659 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:22:53.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:22:53.056844 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-06T00:22:57.255 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:57.308 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:57.308 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 107s 2022-03-06T00:23:01.294 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:01.301678 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:23:01.294 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:01.301698 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:23:01.295 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:01.301706 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:23:01.298 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:01.307796 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:23:01.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:01.439434 I | op-mon: start running mons 2022-03-06T00:23:01.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:01.521153 I | op-mon: creating mon secrets for a new cluster 2022-03-06T00:23:01.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:01.570273 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-06T00:23:01.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:01.573486 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-06T00:23:02.184 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:02.193449 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:23:02.185 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:02.193695 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:23:02.310 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:23:02.438 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:23:02.438 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 112s 2022-03-06T00:23:03.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:03.793926 I | op-mon: targeting the mon count 3 2022-03-06T00:23:03.792 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:03.801764 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-06T00:23:03.815 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:03.823272 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-06T00:23:03.835 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:03.841514 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-06T00:23:05.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.045121 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi018 2022-03-06T00:23:05.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.045207 I | op-mon: mon b assigned to node smithi018 2022-03-06T00:23:05.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.045248 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi059 2022-03-06T00:23:05.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.045287 I | op-mon: mon c assigned to node smithi059 2022-03-06T00:23:05.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.045414 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi019 2022-03-06T00:23:05.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.045430 I | op-mon: mon a assigned to node smithi019 2022-03-06T00:23:05.353 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.362874 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-06T00:23:05.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.552233 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-06T00:23:05.914 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:05.923019 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-06T00:23:06.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.130011 I | op-mon: creating mon a 2022-03-06T00:23:06.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.130042 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:23:06.338 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.346835 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.19:6789"]}] data:a=172.21.15.19:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:-1] 2022-03-06T00:23:06.341 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.347381 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-06T00:23:06.341 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.348256 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-06T00:23:06.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.542926 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:23:06.537 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.543960 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:23:06.693 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.702387 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-06T00:23:06.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:06.842051 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-06T00:23:07.440 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:23:07.650 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:23:07.651 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 117s 2022-03-06T00:23:07.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:07.955539 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.19:6789"]}] data:a=172.21.15.19:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:0] 2022-03-06T00:23:07.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:07.955567 I | op-mon: waiting for mon quorum with [a] 2022-03-06T00:23:08.134 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:08.143988 I | op-mon: mon a is not yet running 2022-03-06T00:23:08.135 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:08.144018 I | op-mon: mons running: [] 2022-03-06T00:23:10.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:10.195742 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789 2022-03-06T00:23:10.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:10.195830 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-06T00:23:10.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:10.195847 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:23:10.192 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:10.197872 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-06T00:23:10.193 INFO:tasks.rook.operator.smithi018.stdout:I0306 00:23:10.197960 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:23:12.654 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:23:12.752 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:23:12.753 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 2m2s 2022-03-06T00:23:17.757 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:23:17.828 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:23:17.828 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 2m7s 2022-03-06T00:23:22.831 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:23:22.897 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:23:22.898 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 2m12s 2022-03-06T00:23:27.900 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:23:27.974 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:23:27.974 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 0/1 ContainerCreating 0 2m17s 2022-03-06T00:23:28.277 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:28.285829 I | op-mon: mons running: [a] 2022-03-06T00:23:32.976 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:23:33.047 INFO:teuthology.orchestra.run.smithi018.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:23:33.047 INFO:teuthology.orchestra.run.smithi018.stdout:rook-ceph-tools-d6d7c985c-hqzf2 1/1 Running 0 2m23s 2022-03-06T00:23:33.049 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-06T00:23:33.049 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:23:48.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:48.472907 I | op-mon: mon a is not yet running 2022-03-06T00:23:48.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:48.472933 I | op-mon: mons running: [] 2022-03-06T00:23:54.562 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:23:54.563 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:23:54.563 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:23:54.563 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:23:54.563 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:23:54.564 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:23:54.564 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:23:54.564 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:23:54.564 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:23:54.565 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:23:54.565 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:23:54.565 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:23:54.565 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:23:54.568 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:23:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:23:54.717 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:54.726884 I | op-mon: Monitors in quorum: [a] 2022-03-06T00:23:54.717 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:54.726904 I | op-mon: mons created: 1 2022-03-06T00:23:54.930 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:54.940109 I | op-mon: waiting for mon quorum with [a] 2022-03-06T00:23:54.984 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:54.994494 I | op-mon: mons running: [a] 2022-03-06T00:23:55.193 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:55.202527 I | op-mon: Monitors in quorum: [a] 2022-03-06T00:23:55.193 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:55.202550 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:23:55.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:55.478107 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:23:55.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:55.478142 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:23:55.789 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:55.798907 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:23:55.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:55.798943 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:23:56.072 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.082206 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:23:56.073 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.082227 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:23:56.347 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.356629 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:23:56.347 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.356679 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:23:56.577 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.586974 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:23:56.578 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.586998 I | op-mon: creating mon b 2022-03-06T00:23:56.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.587003 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:23:56.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.587010 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:23:56.672 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:56.681475 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.19:6789","172.21.15.18:6789"]}] data:a=172.21.15.19:6789,b=172.21.15.18:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:0] 2022-03-06T00:23:57.251 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:57.259368 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:23:57.254 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:57.261830 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:23:57.812 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:57.821785 I | op-mon: 0 of 2 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-06T00:23:57.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:57.826693 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:23:57.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:57.841028 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:23:58.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:58.250466 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-06T00:23:58.580 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:58.589245 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.18:6789","172.21.15.19:6789"]}] data:a=172.21.15.19:6789,b=172.21.15.18:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:1] 2022-03-06T00:23:58.581 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:58.589291 I | op-mon: waiting for mon quorum with [a b] 2022-03-06T00:23:58.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:58.719743 I | op-mon: mon b is not yet running 2022-03-06T00:23:58.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:58.719765 I | op-mon: mons running: [a] 2022-03-06T00:23:58.944 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:58.953761 I | op-mon: Monitors in quorum: [a] 2022-03-06T00:23:58.944 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:58.953790 I | op-mon: mons created: 2 2022-03-06T00:23:59.170 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:59.180051 I | op-mon: waiting for mon quorum with [a b] 2022-03-06T00:23:59.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:59.195783 I | op-mon: mon b is not yet running 2022-03-06T00:23:59.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:23:59.195806 I | op-mon: mons running: [a] 2022-03-06T00:24:04.240 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:04.249907 I | op-mon: mon b is not yet running 2022-03-06T00:24:04.240 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:04.249935 I | op-mon: mons running: [a] 2022-03-06T00:24:04.570 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:24:05.092 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:24:05.093 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:24:05.093 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:24:05.093 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:24:05.093 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:24:05.094 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:24:05.094 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:24:05.094 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:24:05.094 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:24:05.095 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:24:05.095 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:24:05.095 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:24:05.095 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:24:05.099 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:24:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:24:09.257 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:09.267081 I | op-mon: mon b is not yet running 2022-03-06T00:24:09.257 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:09.267111 I | op-mon: mons running: [a] 2022-03-06T00:24:14.275 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:14.284270 I | op-mon: mon b is not yet running 2022-03-06T00:24:14.275 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:14.284297 I | op-mon: mons running: [a] 2022-03-06T00:24:15.102 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:24:15.619 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:24:15.620 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:24:15.620 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:24:15.620 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:24:15.621 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:24:15.621 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:24:15.621 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:24:15.621 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:24:15.622 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:24:15.622 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:24:15.622 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:24:15.622 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:24:15.623 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:24:15.625 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:24:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:24:19.433 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:19.443763 I | op-mon: mon b is not yet running 2022-03-06T00:24:19.434 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:19.443809 I | op-mon: mons running: [a] 2022-03-06T00:24:25.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:25.049292 I | op-mon: mon b is not yet running 2022-03-06T00:24:25.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:25.049319 I | op-mon: mons running: [a] 2022-03-06T00:24:25.628 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:24:26.170 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:24:26.171 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:24:26.171 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:24:26.172 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:24:26.172 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:24:26.172 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:24:26.172 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:24:26.173 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:24:26.173 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:24:26.173 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:24:26.173 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:24:26.174 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:24:26.174 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:24:26.174 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:24:26.177 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:24:30.401 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:30.411027 I | op-mon: mon b is not yet running 2022-03-06T00:24:30.401 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:30.411052 I | op-mon: mons running: [a] 2022-03-06T00:24:35.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:35.423999 I | op-mon: mon b is not yet running 2022-03-06T00:24:35.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:35.424030 I | op-mon: mons running: [a] 2022-03-06T00:24:36.178 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:24:36.697 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:24:36.698 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:24:36.698 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:24:36.699 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:24:36.699 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:24:36.699 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:24:36.699 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:24:36.699 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:24:36.700 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:24:36.700 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:24:36.700 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:24:36.701 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:24:36.701 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:24:36.701 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:24:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:24:40.462 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:40.472131 I | op-mon: mons running: [a b] 2022-03-06T00:24:45.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:45.129684 I | op-mon: Monitors in quorum: [a b] 2022-03-06T00:24:45.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:45.129726 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:24:45.344 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:45.354819 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:24:45.345 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:45.354854 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:24:45.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:45.572746 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:24:45.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:45.572780 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:24:45.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:45.793408 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:24:45.784 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:45.793440 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:24:46.031 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.041684 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:24:46.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.041713 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:24:46.263 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.272979 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:24:46.264 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.273002 I | op-mon: creating mon c 2022-03-06T00:24:46.264 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.273008 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:24:46.264 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.273015 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:24:46.265 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.273020 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:24:46.285 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.295413 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.19:6789","172.21.15.18:6789","172.21.15.59:6789"]}] data:c=172.21.15.59:6789,a=172.21.15.19:6789,b=172.21.15.18:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:1] 2022-03-06T00:24:46.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.342915 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:24:46.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.343185 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:24:46.356 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.366838 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:24:46.371 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.381606 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:24:46.371 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.381640 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:24:46.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.480247 I | op-mon: mons running: [a b c] 2022-03-06T00:24:46.698 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.708532 I | op-mon: Monitors in quorum: [a b] 2022-03-06T00:24:46.704 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.714725 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-06T00:24:46.706 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:24:46.714 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.724804 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-06T00:24:46.714 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:46.724828 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:24:47.071 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:47.081788 I | op-mon: mons running: [a b c] 2022-03-06T00:24:47.201 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:24:47.202 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:24:47.202 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:24:47.202 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:24:47.203 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:24:47.203 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:24:47.203 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:24:47.203 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:24:47.203 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:24:47.204 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:24:47.204 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:24:47.204 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:24:47.204 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:24:47.205 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:24:47.208 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:24:47.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:47.310066 I | op-mon: Monitors in quorum: [a b] 2022-03-06T00:24:47.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:47.345791 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-06T00:24:48.070 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:48.080463 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.19:6789","172.21.15.18:6789","172.21.15.59:6789"]}] data:a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:2] 2022-03-06T00:24:48.070 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:48.080496 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:24:48.673 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:48.683083 I | op-mon: mons running: [a b c] 2022-03-06T00:24:48.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:48.909414 I | op-mon: Monitors in quorum: [a b] 2022-03-06T00:24:48.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:48.909448 I | op-mon: mons created: 3 2022-03-06T00:24:49.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:49.138091 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:24:49.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:49.281685 I | op-mon: mons running: [a b c] 2022-03-06T00:24:49.500 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:49.510999 W | op-mon: failed to find initial monitor c in mon map 2022-03-06T00:24:49.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:49.511018 W | op-mon: monitor c is not in quorum list 2022-03-06T00:24:54.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:54.533511 I | op-mon: mons running: [a b c] 2022-03-06T00:24:54.747 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:54.757525 W | op-mon: failed to find initial monitor c in mon map 2022-03-06T00:24:54.747 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:54.757555 W | op-mon: monitor c is not in quorum list 2022-03-06T00:24:57.210 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:24:57.746 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:24:57.747 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:24:57.747 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:24:57.747 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:24:57.748 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:24:57.748 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:24:57.748 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:24:57.748 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:24:57.749 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:24:57.749 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:24:57.749 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:24:57.749 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:24:57.749 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:24:57.750 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:24:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:24:59.768 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:24:59.778895 I | op-mon: mons running: [a b c] 2022-03-06T00:24:59.996 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:00.006249 W | op-mon: failed to find initial monitor c in mon map 2022-03-06T00:24:59.996 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:00.006281 W | op-mon: monitor c is not in quorum list 2022-03-06T00:25:05.021 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:05.031432 I | op-mon: mons running: [a b c] 2022-03-06T00:25:05.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:05.257124 W | op-mon: failed to find initial monitor c in mon map 2022-03-06T00:25:05.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:05.257151 W | op-mon: monitor c is not in quorum list 2022-03-06T00:25:07.754 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:25:08.252 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:25:08.253 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:25:08.253 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:25:08.253 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:25:08.253 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:25:08.254 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:25:08.254 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:25:08.254 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:25:08.254 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:25:08.255 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:25:08.255 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:25:08.255 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:25:08.255 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:25:08.257 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:25:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:25:10.274 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:10.282281 I | op-mon: mon c is not yet running 2022-03-06T00:25:10.275 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:10.282311 I | op-mon: mons running: [a b] 2022-03-06T00:25:15.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:15.306917 I | op-mon: mon c is not yet running 2022-03-06T00:25:15.297 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:15.306951 I | op-mon: mons running: [a b] 2022-03-06T00:25:18.261 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:25:18.752 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:25:18.752 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:25:18.753 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:25:18.753 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:25:18.753 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:25:18.753 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:25:18.754 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:25:18.754 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:25:18.754 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:25:18.754 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:25:18.754 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:25:18.755 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:25:18.755 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:25:18.756 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:25:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:25:20.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:20.328989 I | op-mon: mon c is not yet running 2022-03-06T00:25:20.319 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:20.329020 I | op-mon: mons running: [a b] 2022-03-06T00:25:25.339 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:25.349714 I | op-mon: mon c is not yet running 2022-03-06T00:25:25.339 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:25.349744 I | op-mon: mons running: [a b] 2022-03-06T00:25:28.761 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:25:29.252 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:25:29.253 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:25:29.253 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:25:29.253 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:25:29.254 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:25:29.254 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:25:29.255 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:25:29.255 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:25:29.256 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:25:29.256 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:25:29.256 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:25:29.257 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:25:29.258 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:25:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:25:29.260 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:25:30.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:30.370978 I | op-mon: mon c is not yet running 2022-03-06T00:25:30.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:30.371004 I | op-mon: mons running: [a b] 2022-03-06T00:25:35.382 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:35.393579 I | op-mon: mon c is not yet running 2022-03-06T00:25:35.383 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:35.393608 I | op-mon: mons running: [a b] 2022-03-06T00:25:39.263 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:25:39.794 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:25:39.795 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:25:39.795 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:25:39.795 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:25:39.795 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:25:39.796 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:25:39.796 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:25:39.796 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:25:39.796 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:25:39.796 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:25:39.797 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:25:39.797 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:25:39.797 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:25:39.798 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:25:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:25:40.404 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:40.414861 I | op-mon: mon c is not yet running 2022-03-06T00:25:40.404 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:40.414893 I | op-mon: mons running: [a b] 2022-03-06T00:25:45.426 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:45.437417 I | op-mon: mon c is not yet running 2022-03-06T00:25:45.427 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:45.437448 I | op-mon: mons running: [a b] 2022-03-06T00:25:49.804 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:25:50.446 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:50.456388 I | op-mon: mons running: [a b c] 2022-03-06T00:25:51.617 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:25:51.618 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:25:51.618 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:25:51.618 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:25:51.619 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:25:51.619 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:25:51.619 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:25:51.619 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:25:51.620 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:25:51.620 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:25:51.620 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:25:51.620 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:25:51.621 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:25:51.624 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:25:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:25:52.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:52.113345 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:25:52.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:52.113384 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:25:52.339 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:52.349912 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:25:52.339 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:52.349934 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:25:52.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:52.579944 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:25:52.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:52.579972 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:25:52.810 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:52.818232 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:25:52.810 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:52.818270 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:25:53.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:53.052967 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:25:53.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:53.052993 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:25:53.262 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:53.273481 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:25:53.263 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:53.273529 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-06T00:25:53.701 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:53.711837 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-06T00:25:54.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:54.270206 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-06T00:25:54.592 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:54.603296 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-06T00:25:54.978 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:54.987988 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-06T00:25:54.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:54.988017 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-06T00:25:55.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:55.311555 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-06T00:25:55.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:55.546661 I | cephclient: successfully enabled msgr2 protocol 2022-03-06T00:25:55.536 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:55.546685 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-06T00:25:55.536 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:55.546689 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-06T00:25:55.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:55.879513 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-06T00:25:55.925 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:55.936104 I | op-mgr: start running mgr 2022-03-06T00:25:55.925 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:55.936133 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-06T00:25:56.671 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:56.681628 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi059": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:25:56.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:25:56.749287 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi059": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:26:01.629 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:26:02.478 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:26:02.479 INFO:teuthology.orchestra.run.smithi018.stderr:no valid command found; 10 closest matches: 2022-03-06T00:26:02.479 INFO:teuthology.orchestra.run.smithi018.stderr:pg stat 2022-03-06T00:26:02.481 INFO:teuthology.orchestra.run.smithi018.stderr:pg getmap 2022-03-06T00:26:02.482 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:26:02.482 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:26:02.482 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_pools_json 2022-03-06T00:26:02.482 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-pool [...] 2022-03-06T00:26:02.483 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-primary [] [...] 2022-03-06T00:26:02.483 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls-by-osd [] [...] 2022-03-06T00:26:02.483 INFO:teuthology.orchestra.run.smithi018.stderr:pg ls [] [...] 2022-03-06T00:26:02.485 INFO:teuthology.orchestra.run.smithi018.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:26:02.486 INFO:teuthology.orchestra.run.smithi018.stderr:Error EINVAL: invalid command 2022-03-06T00:26:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:26:02.488 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 22 2022-03-06T00:26:12.489 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:26:13.048 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:26:13.050 INFO:teuthology.orchestra.run.smithi018.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-06T00:26:13.055 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 2 2022-03-06T00:26:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-06T00:26:17.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.804609 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-06T00:26:17.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.808970 I | op-mgr: setting services to point to mgr "a" 2022-03-06T00:26:17.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.887615 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-06T00:26:17.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.887838 I | op-mgr: successful modules: balancer 2022-03-06T00:26:17.890 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.901247 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-06T00:26:17.890 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.901283 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-06T00:26:17.890 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.901294 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-06T00:26:17.906 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.917392 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-06T00:26:17.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.921157 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-06T00:26:17.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.921187 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-06T00:26:17.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:17.921193 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-06T00:26:18.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:18.359763 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-06T00:26:18.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:18.359804 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:26:18.357 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:18.368706 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-06T00:26:18.357 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:18.368750 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-06T00:26:18.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:18.368766 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-06T00:26:18.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:18.594432 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-06T00:26:18.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:18.595171 I | op-mgr: successful modules: dashboard 2022-03-06T00:26:19.904 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:19.915799 I | op-mgr: successful modules: prometheus 2022-03-06T00:26:23.060 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:26:33.670 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:26:33.673 INFO:teuthology.orchestra.run.smithi018.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-06T00:26:33.677 INFO:teuthology.orchestra.run.smithi018.stderr:command terminated with exit code 2 2022-03-06T00:26:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-06T00:26:33.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:33.845399 I | op-mgr: successful modules: orchestrator modules 2022-03-06T00:26:43.681 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status -f json 2022-03-06T00:26:44.330 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:26:44.330 INFO:teuthology.orchestra.run.smithi018.stdout:{"available": true, "backend": "rook"} 2022-03-06T00:26:44.355 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-06T00:26:44.355 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-06T00:26:44.995 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch apply osd --all-available-devices 2022-03-06T00:26:45.850 INFO:teuthology.orchestra.run.smithi018.stdout:['Success'] 2022-03-06T00:26:45.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.859152 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-06T00:26:45.852 INFO:tasks.rook.operator.smithi018.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", AllowUnsupported: true}, 2022-03-06T00:26:45.852 INFO:tasks.rook.operator.smithi018.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-06T00:26:45.852 INFO:tasks.rook.operator.smithi018.stdout:   ... // 3 identical fields 2022-03-06T00:26:45.852 INFO:tasks.rook.operator.smithi018.stdout:   Config: nil, 2022-03-06T00:26:45.853 INFO:tasks.rook.operator.smithi018.stdout:   Selection: {}, 2022-03-06T00:26:45.853 INFO:tasks.rook.operator.smithi018.stdout:-  StorageClassDeviceSets: nil, 2022-03-06T00:26:45.853 INFO:tasks.rook.operator.smithi018.stdout:+  StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-06T00:26:45.855 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.855 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi018-lv-1", 2022-03-06T00:26:45.855 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.856 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.856 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.856 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.857 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi018-lv-2", 2022-03-06T00:26:45.857 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.857 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.858 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.858 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.859 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi018-lv-3", 2022-03-06T00:26:45.859 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.859 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.860 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.860 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.860 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi018-lv-4", 2022-03-06T00:26:45.860 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.860 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.861 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.865 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.865 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi019-lv-1", 2022-03-06T00:26:45.865 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.865 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.866 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.866 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.867 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi019-lv-2", 2022-03-06T00:26:45.867 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.868 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.869 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.869 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.869 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi019-lv-3", 2022-03-06T00:26:45.869 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.870 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.870 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.870 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.870 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi019-lv-4", 2022-03-06T00:26:45.871 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.871 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.871 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.871 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.872 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi059-lv-1", 2022-03-06T00:26:45.872 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.872 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.872 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.873 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.873 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi059-lv-2", 2022-03-06T00:26:45.873 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.873 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.873 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.874 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.874 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi059-lv-3", 2022-03-06T00:26:45.874 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.874 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.875 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.875 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:26:45.875 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi059-lv-4", 2022-03-06T00:26:45.875 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:26:45.876 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:26:45.876 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.876 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:26:45.876 INFO:tasks.rook.operator.smithi018.stdout:   }, 2022-03-06T00:26:45.876 INFO:tasks.rook.operator.smithi018.stdout:   Annotations: nil, 2022-03-06T00:26:45.877 INFO:tasks.rook.operator.smithi018.stdout:   Labels: nil, 2022-03-06T00:26:45.877 INFO:tasks.rook.operator.smithi018.stdout:   ... // 20 identical fields 2022-03-06T00:26:45.877 INFO:tasks.rook.operator.smithi018.stdout:  } 2022-03-06T00:26:45.877 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.859302 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-06T00:26:45.878 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.859466 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-06T00:26:45.878 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.859565 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-06T00:26:45.878 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.860053 I | operator: successfully started the controller-runtime manager 2022-03-06T00:26:45.878 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.860425 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-06T00:26:45.879 INFO:tasks.rook.operator.smithi018.stdout:I0306 00:26:45.860614 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-06T00:26:45.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.864360 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:26:45.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.864383 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:26:45.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.864405 I | operator: setting up schemes 2022-03-06T00:26:45.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:45.866874 I | operator: setting up the controller-runtime manager 2022-03-06T00:26:45.881 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-06T00:26:45.882 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:26:46.163 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.171713 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:26:46.168 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174426 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:26:46.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174522 I | ceph-cluster-controller: successfully started 2022-03-06T00:26:46.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174613 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:26:46.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174644 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:26:46.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174685 I | ceph-block-pool-controller: successfully started 2022-03-06T00:26:46.170 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174725 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:26:46.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174761 I | ceph-object-realm-controller: successfully started 2022-03-06T00:26:46.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174780 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:26:46.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174794 I | ceph-object-zone-controller: successfully started 2022-03-06T00:26:46.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.174960 I | ceph-object-controller: successfully started 2022-03-06T00:26:46.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175005 I | ceph-file-controller: successfully started 2022-03-06T00:26:46.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175043 I | ceph-nfs-controller: successfully started 2022-03-06T00:26:46.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175074 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:26:46.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175104 I | ceph-client-controller: successfully started 2022-03-06T00:26:46.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175128 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:26:46.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175162 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:26:46.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175184 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:26:46.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175208 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:26:46.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175230 I | ceph-bucket-topic: successfully started 2022-03-06T00:26:46.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175249 I | ceph-bucket-notification: successfully started 2022-03-06T00:26:46.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175269 I | ceph-bucket-notification: successfully started 2022-03-06T00:26:46.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.175298 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:26:46.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.176236 I | operator: starting the controller-runtime manager 2022-03-06T00:26:46.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.284415 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:26:46.277 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.284598 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:26:46.278 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.284661 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:26:46.278 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.284686 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:26:46.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.289788 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:26:46.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.291766 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:26:46.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.292754 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:26:46.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.292818 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-06T00:26:46.286 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.292838 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:26:46.289 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.294884 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-06T00:26:46.289 INFO:tasks.rook.operator.smithi018.stdout:I0306 00:26:46.294982 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:26:46.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.312379 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-06T00:26:46.308 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.317104 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:26:46.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339376 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-06T00:26:46.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339403 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-06T00:26:46.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339411 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-06T00:26:46.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339418 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-06T00:26:46.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339425 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-06T00:26:46.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339432 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-06T00:26:46.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339440 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-06T00:26:46.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339447 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-06T00:26:46.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339454 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-06T00:26:46.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339460 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-06T00:26:46.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339467 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-06T00:26:46.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339474 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-06T00:26:46.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339487 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-06T00:26:46.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339509 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-06T00:26:46.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339522 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-06T00:26:46.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339535 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-06T00:26:46.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339632 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:26:46.337 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:46.339653 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:26:46.479 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:26:46.504 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:26:46.505 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:26:52.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:52.045703 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:26:52.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:52.045737 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:26:52.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:52.045761 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:26:52.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:52.123089 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:26:52.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:52.256459 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:26:52.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:52.256695 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:26:53.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.037807 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-06T00:26:53.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038103 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-06T00:26:53.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038244 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:26:53.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038260 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:26:53.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038271 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:26:53.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038277 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:26:53.035 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038344 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:26:53.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038362 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:26:53.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038369 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:26:53.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038373 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:26:53.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038378 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:26:53.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038382 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:26:53.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038386 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:26:53.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038412 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:26:53.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038425 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-06T00:26:53.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038431 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-06T00:26:53.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038435 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-06T00:26:53.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038494 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-06T00:26:53.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038507 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-06T00:26:53.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038513 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:26:53.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038518 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:26:53.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038523 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-06T00:26:53.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038527 I | ceph-csi: Kubernetes version is 1.23 2022-03-06T00:26:53.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038533 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-06T00:26:53.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.038537 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-06T00:26:53.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.055425 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:26:53.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.102866 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-06T00:26:53.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.111399 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:26:53.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.111419 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:26:53.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.111422 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:26:53.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.111425 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:26:53.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.111427 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:26:53.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.111430 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:26:53.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.111432 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-06T00:26:53.306 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.312012 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:26:53.307 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.312036 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:26:53.307 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.312040 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:26:53.309 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.314354 I | op-mon: start running mons 2022-03-06T00:26:53.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.337353 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:26:53.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.421467 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-06T00:26:53.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.444661 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.59:6789","172.21.15.19:6789","172.21.15.18:6789"]}] data:a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:2] 2022-03-06T00:26:53.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.726271 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:26:53.723 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:53.726575 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:26:54.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:54.284390 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:26:54.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:54.284419 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:26:54.285 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:54.284426 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-06T00:26:54.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:54.799793 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:26:54.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:54.799839 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:26:54.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:54.799860 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:26:54.940 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:54.945612 I | ceph-csi: successfully started CSI CephFS driver 2022-03-06T00:26:56.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.083250 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:26:56.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.088925 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-06T00:26:56.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.088951 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:26:56.090 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.093405 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-06T00:26:56.275 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.280437 I | op-mon: targeting the mon count 3 2022-03-06T00:26:56.280 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.286158 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:26:56.507 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:26:56.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.637869 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:26:56.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.637900 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:26:56.976 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.981390 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:26:57.168 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:56.981424 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:26:57.170 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:26:57.171 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:26:57.171 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:26:57.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:57.321768 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:26:57.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:57.321805 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:26:57.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:57.664356 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:26:57.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:57.664380 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:26:58.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.007107 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:26:58.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.007142 I | op-mon: checking for basic quorum with existing mons 2022-03-06T00:26:58.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.007156 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:26:58.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.007169 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:26:58.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.007179 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:26:58.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.044981 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.59:6789","172.21.15.19:6789","172.21.15.18:6789"]}] data:a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:2] 2022-03-06T00:26:58.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.082440 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:26:58.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.082710 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:26:58.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.101690 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:26:58.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.109986 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:26:58.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.110012 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:26:58.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.484853 I | op-mon: mons running: [a b c] 2022-03-06T00:26:58.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.907054 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:26:58.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.914463 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-06T00:26:58.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.928240 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-06T00:26:58.924 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:58.928268 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:26:59.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:59.085698 I | op-mon: mons running: [a b c] 2022-03-06T00:26:59.534 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:59.538975 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:26:59.542 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:59.547314 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-06T00:26:59.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:59.555026 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-06T00:26:59.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:59.555047 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:26:59.680 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:26:59.684786 I | op-mon: mons running: [a b c] 2022-03-06T00:27:00.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:00.106258 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:00.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:00.106311 I | op-mon: mons created: 3 2022-03-06T00:27:00.544 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:00.549193 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:00.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:00.566476 I | op-mon: mons running: [a b c] 2022-03-06T00:27:00.968 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:00.972511 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:00.969 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:00.972543 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-06T00:27:01.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:01.395282 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-06T00:27:01.818 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:01.822598 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-06T00:27:02.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:02.237231 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-06T00:27:02.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:02.769768 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-06T00:27:02.766 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:02.769795 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-06T00:27:03.229 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:03.232927 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-06T00:27:03.648 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:03.651172 I | cephclient: successfully enabled msgr2 protocol 2022-03-06T00:27:03.648 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:03.651212 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-06T00:27:03.648 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:03.651226 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-06T00:27:04.071 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.075128 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-06T00:27:04.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.098977 I | op-mgr: start running mgr 2022-03-06T00:27:04.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.099002 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-06T00:27:04.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.596919 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-06T00:27:04.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.608993 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-06T00:27:04.608 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.612092 I | op-mgr: setting services to point to mgr "a" 2022-03-06T00:27:04.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.754061 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-06T00:27:04.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.754202 I | op-mgr: successful modules: balancer 2022-03-06T00:27:04.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.771932 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-06T00:27:04.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.771958 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-06T00:27:04.786 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.790431 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-06T00:27:04.948 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:04.952307 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-1" 2022-03-06T00:27:05.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.157063 I | op-osd: successfully provisioned PVC "smithi018-lv-1-data-0j686x" 2022-03-06T00:27:05.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.157085 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-2" 2022-03-06T00:27:05.353 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.356661 I | op-osd: successfully provisioned PVC "smithi018-lv-2-data-0jmljt" 2022-03-06T00:27:05.354 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.356713 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-3" 2022-03-06T00:27:05.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.555621 I | op-osd: successfully provisioned PVC "smithi018-lv-3-data-0t695v" 2022-03-06T00:27:05.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.555666 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-4" 2022-03-06T00:27:05.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.756049 I | op-osd: successfully provisioned PVC "smithi018-lv-4-data-0qhqwb" 2022-03-06T00:27:05.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.756101 I | op-osd: creating 1 new PVCs for device set "smithi019-lv-1" 2022-03-06T00:27:05.953 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.956866 I | op-osd: successfully provisioned PVC "smithi019-lv-1-data-0n4sw2" 2022-03-06T00:27:05.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:05.956910 I | op-osd: creating 1 new PVCs for device set "smithi019-lv-2" 2022-03-06T00:27:06.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.156190 I | op-osd: successfully provisioned PVC "smithi019-lv-2-data-0dgfhd" 2022-03-06T00:27:06.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.156229 I | op-osd: creating 1 new PVCs for device set "smithi019-lv-3" 2022-03-06T00:27:06.181 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.184285 I | op-mgr: successful modules: dashboard 2022-03-06T00:27:06.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.190759 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-06T00:27:06.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.191180 I | op-mgr: successful modules: prometheus 2022-03-06T00:27:06.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.354616 I | op-osd: successfully provisioned PVC "smithi019-lv-3-data-08q647" 2022-03-06T00:27:06.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.354648 I | op-osd: creating 1 new PVCs for device set "smithi019-lv-4" 2022-03-06T00:27:06.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.555791 I | op-osd: successfully provisioned PVC "smithi019-lv-4-data-0t7rhl" 2022-03-06T00:27:06.554 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.555829 I | op-osd: creating 1 new PVCs for device set "smithi059-lv-1" 2022-03-06T00:27:06.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.756083 I | op-osd: successfully provisioned PVC "smithi059-lv-1-data-0js9sd" 2022-03-06T00:27:06.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.756133 I | op-osd: creating 1 new PVCs for device set "smithi059-lv-2" 2022-03-06T00:27:06.953 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.956074 I | op-osd: successfully provisioned PVC "smithi059-lv-2-data-0knbvw" 2022-03-06T00:27:06.953 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:06.956134 I | op-osd: creating 1 new PVCs for device set "smithi059-lv-3" 2022-03-06T00:27:07.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:07.156312 I | op-osd: successfully provisioned PVC "smithi059-lv-3-data-0fp8vd" 2022-03-06T00:27:07.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:07.156373 I | op-osd: creating 1 new PVCs for device set "smithi059-lv-4" 2022-03-06T00:27:07.172 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:27:07.354 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:07.356358 I | op-osd: successfully provisioned PVC "smithi059-lv-4-data-0jq895" 2022-03-06T00:27:07.367 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:07.370169 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-0j686x" 2022-03-06T00:27:07.705 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:27:07.724 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:27:07.725 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:27:07.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:07.775262 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-1-data-0j686x" 2022-03-06T00:27:07.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:07.775297 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0jmljt" 2022-03-06T00:27:08.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:08.002395 I | op-mgr: successful modules: orchestrator modules 2022-03-06T00:27:08.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:08.190848 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-2-data-0jmljt" 2022-03-06T00:27:08.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:08.190875 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0t695v" 2022-03-06T00:27:08.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:08.645260 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-3-data-0t695v" 2022-03-06T00:27:08.643 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:08.645299 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0qhqwb" 2022-03-06T00:27:09.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:09.312678 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-4-data-0qhqwb" 2022-03-06T00:27:09.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:09.312701 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-1-data-0n4sw2" 2022-03-06T00:27:09.641 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:09.643972 I | op-osd: started OSD provisioning job for PVC "smithi019-lv-1-data-0n4sw2" 2022-03-06T00:27:09.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:09.644016 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-2-data-0dgfhd" 2022-03-06T00:27:10.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:10.176892 I | op-osd: started OSD provisioning job for PVC "smithi019-lv-2-data-0dgfhd" 2022-03-06T00:27:10.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:10.176919 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-3-data-08q647" 2022-03-06T00:27:10.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:10.395291 I | op-osd: started OSD provisioning job for PVC "smithi019-lv-3-data-08q647" 2022-03-06T00:27:10.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:10.395317 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-4-data-0t7rhl" 2022-03-06T00:27:10.572 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:10.575058 I | op-osd: started OSD provisioning job for PVC "smithi019-lv-4-data-0t7rhl" 2022-03-06T00:27:10.573 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:10.575086 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-1-data-0js9sd" 2022-03-06T00:27:11.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:11.202654 I | op-osd: started OSD provisioning job for PVC "smithi059-lv-1-data-0js9sd" 2022-03-06T00:27:11.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:11.202689 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-2-data-0knbvw" 2022-03-06T00:27:11.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:11.663098 I | op-osd: started OSD provisioning job for PVC "smithi059-lv-2-data-0knbvw" 2022-03-06T00:27:11.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:11.663123 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-3-data-0fp8vd" 2022-03-06T00:27:11.901 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:11.903443 I | op-osd: started OSD provisioning job for PVC "smithi059-lv-3-data-0fp8vd" 2022-03-06T00:27:11.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:11.903474 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-4-data-0jq895" 2022-03-06T00:27:12.875 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.876656 I | op-osd: started OSD provisioning job for PVC "smithi059-lv-4-data-0jq895" 2022-03-06T00:27:12.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.876721 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-06T00:27:12.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.876742 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-06T00:27:12.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880694 I | op-osd: OSD orchestration status for PVC smithi018-lv-1-data-0j686x is "starting" 2022-03-06T00:27:12.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880719 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0jmljt is "starting" 2022-03-06T00:27:12.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880724 I | op-osd: OSD orchestration status for PVC smithi018-lv-3-data-0t695v is "starting" 2022-03-06T00:27:12.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880729 I | op-osd: OSD orchestration status for PVC smithi018-lv-4-data-0qhqwb is "starting" 2022-03-06T00:27:12.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880734 I | op-osd: OSD orchestration status for PVC smithi019-lv-1-data-0n4sw2 is "starting" 2022-03-06T00:27:12.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880742 I | op-osd: OSD orchestration status for PVC smithi019-lv-2-data-0dgfhd is "starting" 2022-03-06T00:27:12.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880752 I | op-osd: OSD orchestration status for PVC smithi019-lv-3-data-08q647 is "starting" 2022-03-06T00:27:12.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880757 I | op-osd: OSD orchestration status for PVC smithi019-lv-4-data-0t7rhl is "starting" 2022-03-06T00:27:12.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880762 I | op-osd: OSD orchestration status for PVC smithi059-lv-1-data-0js9sd is "starting" 2022-03-06T00:27:12.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880767 I | op-osd: OSD orchestration status for PVC smithi059-lv-2-data-0knbvw is "starting" 2022-03-06T00:27:12.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880771 I | op-osd: OSD orchestration status for PVC smithi059-lv-3-data-0fp8vd is "starting" 2022-03-06T00:27:12.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:12.880777 I | op-osd: OSD orchestration status for PVC smithi059-lv-4-data-0jq895 is "starting" 2022-03-06T00:27:16.421 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:16.422733 I | op-osd: OSD orchestration status for node smithi018-lv-2-data-0jmljt is "orchestrating" 2022-03-06T00:27:17.490 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:17.491172 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0jmljt is "orchestrating" 2022-03-06T00:27:17.726 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:27:17.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:17.966577 I | op-osd: OSD orchestration status for node smithi019-lv-1-data-0n4sw2 is "orchestrating" 2022-03-06T00:27:18.343 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:18.344302 I | op-osd: OSD orchestration status for PVC smithi019-lv-1-data-0n4sw2 is "orchestrating" 2022-03-06T00:27:18.441 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:27:18.458 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":3,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:27:18.459 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:27:25.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:25.226312 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0jmljt is "completed" 2022-03-06T00:27:25.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:25.226339 I | op-osd: creating OSD 0 on PVC "smithi018-lv-2-data-0jmljt" 2022-03-06T00:27:25.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:25.226347 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0jmljt" 2022-03-06T00:27:26.510 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:26.510704 I | op-osd: OSD orchestration status for node smithi059-lv-1-data-0js9sd is "orchestrating" 2022-03-06T00:27:26.515 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:26.511091 I | op-osd: OSD orchestration status for node smithi059-lv-2-data-0knbvw is "orchestrating" 2022-03-06T00:27:26.515 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:26.511215 I | op-osd: OSD orchestration status for PVC smithi059-lv-1-data-0js9sd is "orchestrating" 2022-03-06T00:27:26.515 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:26.511684 I | op-osd: OSD orchestration status for node smithi059-lv-3-data-0fp8vd is "orchestrating" 2022-03-06T00:27:26.516 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:26.512066 I | op-osd: OSD orchestration status for PVC smithi059-lv-2-data-0knbvw is "orchestrating" 2022-03-06T00:27:26.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:26.889238 I | op-osd: OSD orchestration status for PVC smithi059-lv-3-data-0fp8vd is "orchestrating" 2022-03-06T00:27:28.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:28.234345 I | op-osd: OSD orchestration status for PVC smithi019-lv-1-data-0n4sw2 is "completed" 2022-03-06T00:27:29.132 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:28.234426 I | op-osd: creating OSD 1 on PVC "smithi019-lv-1-data-0n4sw2" 2022-03-06T00:27:29.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:28.234464 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-1-data-0n4sw2" 2022-03-06T00:27:29.133 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:27:29.905 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:29.904568 I | op-osd: OSD orchestration status for node smithi018-lv-1-data-0j686x is "orchestrating" 2022-03-06T00:27:29.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:29.908314 I | op-osd: OSD orchestration status for PVC smithi018-lv-1-data-0j686x is "orchestrating" 2022-03-06T00:27:30.251 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:27:30.269 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":5,"num_osds":2,"num_up_osds":0,"osd_up_since":0,"num_in_osds":2,"osd_in_since":1646526440,"num_remapped_pgs":0} 2022-03-06T00:27:30.270 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:27:30.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:30.345642 I | op-osd: OSD orchestration status for node smithi059-lv-4-data-0jq895 is "orchestrating" 2022-03-06T00:27:31.252 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:31.251999 I | op-osd: OSD orchestration status for PVC smithi059-lv-4-data-0jq895 is "orchestrating" 2022-03-06T00:27:34.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:34.457820 I | op-osd: OSD orchestration status for node smithi018-lv-3-data-0t695v is "orchestrating" 2022-03-06T00:27:34.986 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:34.985087 I | op-osd: OSD orchestration status for PVC smithi018-lv-3-data-0t695v is "orchestrating" 2022-03-06T00:27:35.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:35.734407 I | op-osd: OSD orchestration status for node smithi019-lv-4-data-0t7rhl is "orchestrating" 2022-03-06T00:27:35.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:35.762545 I | op-osd: OSD orchestration status for PVC smithi059-lv-1-data-0js9sd is "completed" 2022-03-06T00:27:35.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:35.762581 I | op-osd: creating OSD 2 on PVC "smithi059-lv-1-data-0js9sd" 2022-03-06T00:27:35.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:35.762598 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-1-data-0js9sd" 2022-03-06T00:27:36.913 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:36.912093 I | op-osd: OSD orchestration status for PVC smithi019-lv-4-data-0t7rhl is "orchestrating" 2022-03-06T00:27:36.913 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:36.912238 I | op-osd: OSD orchestration status for node smithi018-lv-4-data-0qhqwb is "orchestrating" 2022-03-06T00:27:37.101 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:37.100730 I | op-osd: OSD orchestration status for PVC smithi059-lv-2-data-0knbvw is "completed" 2022-03-06T00:27:37.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:37.100758 I | op-osd: creating OSD 3 on PVC "smithi059-lv-2-data-0knbvw" 2022-03-06T00:27:37.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:37.100772 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-2-data-0knbvw" 2022-03-06T00:27:37.266 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:37.265529 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi059": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:27:37.605 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:37.604300 I | op-osd: OSD orchestration status for PVC smithi018-lv-4-data-0qhqwb is "orchestrating" 2022-03-06T00:27:37.771 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:37.770173 I | op-osd: OSD orchestration status for node smithi019-lv-2-data-0dgfhd is "orchestrating" 2022-03-06T00:27:38.027 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:38.025861 I | op-osd: OSD orchestration status for node smithi019-lv-3-data-08q647 is "orchestrating" 2022-03-06T00:27:38.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:38.662286 I | op-osd: OSD orchestration status for PVC smithi019-lv-2-data-0dgfhd is "orchestrating" 2022-03-06T00:27:38.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:38.662354 I | op-osd: OSD orchestration status for PVC smithi059-lv-3-data-0fp8vd is "completed" 2022-03-06T00:27:38.664 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:38.662373 I | op-osd: creating OSD 4 on PVC "smithi059-lv-3-data-0fp8vd" 2022-03-06T00:27:38.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:38.662387 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-3-data-0fp8vd" 2022-03-06T00:27:39.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:39.004773 I | op-osd: OSD orchestration status for PVC smithi019-lv-3-data-08q647 is "orchestrating" 2022-03-06T00:27:39.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:39.005181 I | op-osd: OSD orchestration status for PVC smithi018-lv-1-data-0j686x is "completed" 2022-03-06T00:27:39.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:39.005196 I | op-osd: creating OSD 5 on PVC "smithi018-lv-1-data-0j686x" 2022-03-06T00:27:39.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:39.005204 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-0j686x" 2022-03-06T00:27:40.272 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:27:41.636 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:27:41.656 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":13,"num_osds":10,"num_up_osds":0,"osd_up_since":0,"num_in_osds":10,"osd_in_since":1646526460,"num_remapped_pgs":0} 2022-03-06T00:27:41.656 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:27:43.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:43.034551 I | op-osd: OSD orchestration status for PVC smithi059-lv-4-data-0jq895 is "completed" 2022-03-06T00:27:43.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:43.034583 I | op-osd: creating OSD 6 on PVC "smithi059-lv-4-data-0jq895" 2022-03-06T00:27:43.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:43.034602 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-4-data-0jq895" 2022-03-06T00:27:45.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:45.095840 I | op-osd: OSD orchestration status for PVC smithi018-lv-3-data-0t695v is "completed" 2022-03-06T00:27:45.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:45.095863 I | op-osd: creating OSD 7 on PVC "smithi018-lv-3-data-0t695v" 2022-03-06T00:27:45.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:45.095870 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0t695v" 2022-03-06T00:27:46.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:46.638552 I | op-osd: OSD orchestration status for PVC smithi019-lv-4-data-0t7rhl is "completed" 2022-03-06T00:27:46.643 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:46.638592 I | op-osd: creating OSD 8 on PVC "smithi019-lv-4-data-0t7rhl" 2022-03-06T00:27:46.643 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:46.638608 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-4-data-0t7rhl" 2022-03-06T00:27:46.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:46.867538 I | op-osd: OSD orchestration status for PVC smithi019-lv-2-data-0dgfhd is "completed" 2022-03-06T00:27:46.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:46.867587 I | op-osd: creating OSD 9 on PVC "smithi019-lv-2-data-0dgfhd" 2022-03-06T00:27:46.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:46.867609 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-2-data-0dgfhd" 2022-03-06T00:27:48.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:48.780905 I | op-osd: OSD orchestration status for PVC smithi019-lv-3-data-08q647 is "completed" 2022-03-06T00:27:48.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:48.780952 I | op-osd: creating OSD 11 on PVC "smithi019-lv-3-data-08q647" 2022-03-06T00:27:48.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:48.780963 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-3-data-08q647" 2022-03-06T00:27:50.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:50.214419 I | op-osd: OSD orchestration status for PVC smithi018-lv-4-data-0qhqwb is "completed" 2022-03-06T00:27:50.218 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:50.214482 I | op-osd: creating OSD 10 on PVC "smithi018-lv-4-data-0qhqwb" 2022-03-06T00:27:50.218 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:50.214499 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0qhqwb" 2022-03-06T00:27:51.659 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:27:52.900 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:27:52.918 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":16,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646526462,"num_remapped_pgs":0} 2022-03-06T00:27:52.918 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:27:53.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:53.059029 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-06T00:27:53.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:27:53.059062 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:28:02.919 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:28:03.596 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:03.615 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":20,"num_osds":12,"num_up_osds":2,"osd_up_since":1646526476,"num_in_osds":12,"osd_in_since":1646526462,"num_remapped_pgs":0} 2022-03-06T00:28:03.616 INFO:tasks.rook: have 2/12 OSDs 2022-03-06T00:28:13.617 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:28:14.201 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:14.225 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":25,"num_osds":12,"num_up_osds":4,"osd_up_since":1646526493,"num_in_osds":12,"osd_in_since":1646526462,"num_remapped_pgs":0} 2022-03-06T00:28:14.226 INFO:tasks.rook: have 4/12 OSDs 2022-03-06T00:28:24.228 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd stat -f json 2022-03-06T00:28:24.803 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:24.822 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":30,"num_osds":12,"num_up_osds":12,"osd_up_since":1646526497,"num_in_osds":12,"osd_in_since":1646526462,"num_remapped_pgs":0} 2022-03-06T00:28:24.823 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-06T00:28:24.824 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- cat /etc/ceph/ceph.conf 2022-03-06T00:28:25.049 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2022-03-06T00:28:25.049 INFO:teuthology.orchestra.run.smithi018.stdout:mon_host = 172.21.15.19:6789,172.21.15.18:6789,172.21.15.59:6789 2022-03-06T00:28:25.050 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:25.050 INFO:teuthology.orchestra.run.smithi018.stdout:[client.admin] 2022-03-06T00:28:25.050 INFO:teuthology.orchestra.run.smithi018.stdout:keyring = /etc/ceph/keyring 2022-03-06T00:28:25.057 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- cat /etc/ceph/keyring 2022-03-06T00:28:25.238 INFO:teuthology.orchestra.run.smithi018.stdout:[client.admin] 2022-03-06T00:28:25.239 INFO:teuthology.orchestra.run.smithi018.stdout:key = AQDl/iNikwH7HhAAtC4J04qGEc6G7Nn6hffv+A== 2022-03-06T00:28:25.243 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph 2022-03-06T00:28:25.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph 2022-03-06T00:28:25.271 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph 2022-03-06T00:28:25.284 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:28:25.285 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/ceph.conf 2022-03-06T00:28:25.312 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:28:25.312 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/keyring 2022-03-06T00:28:25.324 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:28:25.325 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.conf 2022-03-06T00:28:25.341 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:28:25.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/keyring 2022-03-06T00:28:25.423 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:28:25.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.conf 2022-03-06T00:28:25.441 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:28:25.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/keyring 2022-03-06T00:28:25.457 INFO:tasks.rook:Setting up client nodes... 2022-03-06T00:28:25.457 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-06T00:28:26.314 INFO:teuthology.orchestra.run.smithi018.stdout:[client.a] 2022-03-06T00:28:26.314 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAqACRixmTIAxAAjwhn0rd++TFZ2KP+vkZn7g== 2022-03-06T00:28:26.335 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2022-03-06T00:28:26.335 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-06T00:28:26.336 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-06T00:28:26.362 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-06T00:28:27.055 INFO:teuthology.orchestra.run.smithi018.stdout:[client.b] 2022-03-06T00:28:27.055 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAqACRiGLRhORAAQQWSGj6Cxp+Bmk2WySyzWw== 2022-03-06T00:28:27.079 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2022-03-06T00:28:27.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-06T00:28:27.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-06T00:28:27.102 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-06T00:28:27.908 INFO:teuthology.orchestra.run.smithi018.stdout:[client.c] 2022-03-06T00:28:27.909 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQArACRiCC2ULxAAiKEpzNwmPgWF1PhHua7H9g== 2022-03-06T00:28:27.939 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-03-06T00:28:27.940 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-06T00:28:27.940 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-06T00:28:27.965 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd dump --format=json 2022-03-06T00:28:28.504 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:28.504 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":30,"fsid":"509894aa-97af-4ae5-a89a-865199c2e73d","created":"2022-03-06T00:23:53.351658+0000","modified":"2022-03-06T00:28:19.475543+0000","last_up_change":"2022-03-06T00:28:17.921325+0000","last_in_change":"2022-03-06T00:27:42.604573+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-06T00:28:13.063439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"37072002-518c-46e6-a446-34d8b1623080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6801","nonce":3083061169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6803","nonce":3083061169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6807","nonce":3083061169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6805","nonce":3083061169}]},"public_addr":"172.21.15.18:6801/3083061169","cluster_addr":"172.21.15.18:6803/3083061169","heartbeat_back_addr":"172.21.15.18:6807/3083061169","heartbeat_front_addr":"172.21.15.18:6805/3083061169","state":["exists","up"]},{"osd":1,"uuid":"ca74c2b9-c7f6-4694-8e9a-be3abe3d6859","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6801","nonce":4012638484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6803","nonce":4012638484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6807","nonce":4012638484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6805","nonce":4012638484}]},"public_addr":"172.21.15.19:6801/4012638484","cluster_addr":"172.21.15.19:6803/4012638484","heartbeat_back_addr":"172.21.15.19:6807/4012638484","heartbeat_front_addr":"172.21.15.19:6805/4012638484","state":["exists","up"]},{"osd":2,"uuid":"6fb19e44-feef-4b43-be82-0e1228ac862d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6819","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6821","nonce":1625184900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6823","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6825","nonce":1625184900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6831","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6833","nonce":1625184900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6829","nonce":1625184900}]},"public_addr":"172.21.15.59:6821/1625184900","cluster_addr":"172.21.15.59:6825/1625184900","heartbeat_back_addr":"172.21.15.59:6833/1625184900","heartbeat_front_addr":"172.21.15.59:6829/1625184900","state":["exists","up"]},{"osd":3,"uuid":"5cb8fc2b-399c-4f4d-8468-413d140ddab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6811","nonce":3422330840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6813","nonce":3422330840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6817","nonce":3422330840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6815","nonce":3422330840}]},"public_addr":"172.21.15.59:6811/3422330840","cluster_addr":"172.21.15.59:6813/3422330840","heartbeat_back_addr":"172.21.15.59:6817/3422330840","heartbeat_front_addr":"172.21.15.59:6815/3422330840","state":["exists","up"]},{"osd":4,"uuid":"cd401e96-7f2b-443e-b113-8827d4b0081f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6820","nonce":3444139581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6824","nonce":3444139581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6832","nonce":3444139581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6827","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6828","nonce":3444139581}]},"public_addr":"172.21.15.59:6820/3444139581","cluster_addr":"172.21.15.59:6824/3444139581","heartbeat_back_addr":"172.21.15.59:6832/3444139581","heartbeat_front_addr":"172.21.15.59:6828/3444139581","state":["exists","up"]},{"osd":5,"uuid":"2ca11032-ca69-405a-a237-88a110504873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6809","nonce":2660541088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6811","nonce":2660541088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6815","nonce":2660541088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6812","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6813","nonce":2660541088}]},"public_addr":"172.21.15.18:6809/2660541088","cluster_addr":"172.21.15.18:6811/2660541088","heartbeat_back_addr":"172.21.15.18:6815/2660541088","heartbeat_front_addr":"172.21.15.18:6813/2660541088","state":["exists","up"]},{"osd":6,"uuid":"364afe94-5e85-4316-b8ec-470c170ce7ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6803","nonce":516578741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6805","nonce":516578741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6809","nonce":516578741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6807","nonce":516578741}]},"public_addr":"172.21.15.59:6803/516578741","cluster_addr":"172.21.15.59:6805/516578741","heartbeat_back_addr":"172.21.15.59:6809/516578741","heartbeat_front_addr":"172.21.15.59:6807/516578741","state":["exists","up"]},{"osd":7,"uuid":"ff8b1be4-3cef-4410-8c31-c5e815c52acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6824","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6825","nonce":2724201665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6826","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6827","nonce":2724201665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6830","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6831","nonce":2724201665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6828","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6829","nonce":2724201665}]},"public_addr":"172.21.15.18:6825/2724201665","cluster_addr":"172.21.15.18:6827/2724201665","heartbeat_back_addr":"172.21.15.18:6831/2724201665","heartbeat_front_addr":"172.21.15.18:6829/2724201665","state":["exists","up"]},{"osd":8,"uuid":"2c2c6946-fab1-47d5-a8ed-c7b76b22ea3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6809","nonce":3504998958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6811","nonce":3504998958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6815","nonce":3504998958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6813","nonce":3504998958}]},"public_addr":"172.21.15.19:6809/3504998958","cluster_addr":"172.21.15.19:6811/3504998958","heartbeat_back_addr":"172.21.15.19:6815/3504998958","heartbeat_front_addr":"172.21.15.19:6813/3504998958","state":["exists","up"]},{"osd":9,"uuid":"ecc63a20-91fd-491b-ae24-1987cb89b79e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6825","nonce":659409981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6827","nonce":659409981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6831","nonce":659409981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6829","nonce":659409981}]},"public_addr":"172.21.15.19:6825/659409981","cluster_addr":"172.21.15.19:6827/659409981","heartbeat_back_addr":"172.21.15.19:6831/659409981","heartbeat_front_addr":"172.21.15.19:6829/659409981","state":["exists","up"]},{"osd":10,"uuid":"dab2c556-8182-40c4-8905-ed4d3f3eef0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6816","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6817","nonce":3054655181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6819","nonce":3054655181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6823","nonce":3054655181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6820","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6821","nonce":3054655181}]},"public_addr":"172.21.15.18:6817/3054655181","cluster_addr":"172.21.15.18:6819/3054655181","heartbeat_back_addr":"172.21.15.18:6823/3054655181","heartbeat_front_addr":"172.21.15.18:6821/3054655181","state":["exists","up"]},{"osd":11,"uuid":"6ad55028-2074-457d-9de8-83e2194b4ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6817","nonce":3207016817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6819","nonce":3207016817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6823","nonce":3207016817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6821","nonce":3207016817}]},"public_addr":"172.21.15.19:6817/3207016817","cluster_addr":"172.21.15.19:6819/3207016817","heartbeat_back_addr":"172.21.15.19:6823/3207016817","heartbeat_front_addr":"172.21.15.19:6821/3207016817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:27:49.829069+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:27:50.502721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.459409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.385240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.522717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:07.668464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:11.401425+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.247625+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.184580+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:15.874632+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.205918+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:15.833881+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2163995605":"2022-03-07T00:26:25.217327+0000","172.21.15.59:0/2251141036":"2022-03-07T00:26:25.217327+0000","172.21.15.59:6801/2847001190":"2022-03-07T00:26:25.217327+0000","172.21.15.59:6800/2847001190":"2022-03-07T00:26:25.217327+0000","172.21.15.59:0/1767239452":"2022-03-07T00:26:25.217327+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-06T00:28:28.524 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-06T00:28:13.063439+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-06T00:28:28.524 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd pool get .mgr pg_num 2022-03-06T00:28:29.259 INFO:teuthology.orchestra.run.smithi018.stdout:pg_num: 1 2022-03-06T00:28:29.280 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-06T00:28:29.281 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-06T00:28:29.282 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph mgr dump --format=json 2022-03-06T00:28:29.858 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:29.878 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":13,"active_gid":24206,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1909005709},{"type":"v1","addr":"172.21.15.59:6801","nonce":1909005709}]},"active_addr":"172.21.15.59:6801/1909005709","active_change":"2022-03-06T00:26:32.252970+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://172.21.15.59:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":2004768381}]},{"addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":4203309967}]},{"addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":1826480473}]},{"addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":2196089222}]}]}} 2022-03-06T00:28:29.882 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-06T00:28:29.883 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-06T00:28:29.883 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd dump --format=json 2022-03-06T00:28:30.407 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:30.408 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":30,"fsid":"509894aa-97af-4ae5-a89a-865199c2e73d","created":"2022-03-06T00:23:53.351658+0000","modified":"2022-03-06T00:28:19.475543+0000","last_up_change":"2022-03-06T00:28:17.921325+0000","last_in_change":"2022-03-06T00:27:42.604573+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-06T00:28:13.063439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"37072002-518c-46e6-a446-34d8b1623080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6801","nonce":3083061169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6803","nonce":3083061169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6807","nonce":3083061169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6805","nonce":3083061169}]},"public_addr":"172.21.15.18:6801/3083061169","cluster_addr":"172.21.15.18:6803/3083061169","heartbeat_back_addr":"172.21.15.18:6807/3083061169","heartbeat_front_addr":"172.21.15.18:6805/3083061169","state":["exists","up"]},{"osd":1,"uuid":"ca74c2b9-c7f6-4694-8e9a-be3abe3d6859","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6801","nonce":4012638484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6803","nonce":4012638484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6807","nonce":4012638484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6805","nonce":4012638484}]},"public_addr":"172.21.15.19:6801/4012638484","cluster_addr":"172.21.15.19:6803/4012638484","heartbeat_back_addr":"172.21.15.19:6807/4012638484","heartbeat_front_addr":"172.21.15.19:6805/4012638484","state":["exists","up"]},{"osd":2,"uuid":"6fb19e44-feef-4b43-be82-0e1228ac862d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6819","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6821","nonce":1625184900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6823","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6825","nonce":1625184900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6831","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6833","nonce":1625184900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6829","nonce":1625184900}]},"public_addr":"172.21.15.59:6821/1625184900","cluster_addr":"172.21.15.59:6825/1625184900","heartbeat_back_addr":"172.21.15.59:6833/1625184900","heartbeat_front_addr":"172.21.15.59:6829/1625184900","state":["exists","up"]},{"osd":3,"uuid":"5cb8fc2b-399c-4f4d-8468-413d140ddab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6811","nonce":3422330840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6813","nonce":3422330840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6817","nonce":3422330840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6815","nonce":3422330840}]},"public_addr":"172.21.15.59:6811/3422330840","cluster_addr":"172.21.15.59:6813/3422330840","heartbeat_back_addr":"172.21.15.59:6817/3422330840","heartbeat_front_addr":"172.21.15.59:6815/3422330840","state":["exists","up"]},{"osd":4,"uuid":"cd401e96-7f2b-443e-b113-8827d4b0081f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6820","nonce":3444139581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6824","nonce":3444139581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6832","nonce":3444139581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6827","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6828","nonce":3444139581}]},"public_addr":"172.21.15.59:6820/3444139581","cluster_addr":"172.21.15.59:6824/3444139581","heartbeat_back_addr":"172.21.15.59:6832/3444139581","heartbeat_front_addr":"172.21.15.59:6828/3444139581","state":["exists","up"]},{"osd":5,"uuid":"2ca11032-ca69-405a-a237-88a110504873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6809","nonce":2660541088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6811","nonce":2660541088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6815","nonce":2660541088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6812","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6813","nonce":2660541088}]},"public_addr":"172.21.15.18:6809/2660541088","cluster_addr":"172.21.15.18:6811/2660541088","heartbeat_back_addr":"172.21.15.18:6815/2660541088","heartbeat_front_addr":"172.21.15.18:6813/2660541088","state":["exists","up"]},{"osd":6,"uuid":"364afe94-5e85-4316-b8ec-470c170ce7ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6803","nonce":516578741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6805","nonce":516578741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6809","nonce":516578741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6807","nonce":516578741}]},"public_addr":"172.21.15.59:6803/516578741","cluster_addr":"172.21.15.59:6805/516578741","heartbeat_back_addr":"172.21.15.59:6809/516578741","heartbeat_front_addr":"172.21.15.59:6807/516578741","state":["exists","up"]},{"osd":7,"uuid":"ff8b1be4-3cef-4410-8c31-c5e815c52acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6824","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6825","nonce":2724201665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6826","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6827","nonce":2724201665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6830","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6831","nonce":2724201665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6828","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6829","nonce":2724201665}]},"public_addr":"172.21.15.18:6825/2724201665","cluster_addr":"172.21.15.18:6827/2724201665","heartbeat_back_addr":"172.21.15.18:6831/2724201665","heartbeat_front_addr":"172.21.15.18:6829/2724201665","state":["exists","up"]},{"osd":8,"uuid":"2c2c6946-fab1-47d5-a8ed-c7b76b22ea3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6809","nonce":3504998958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6811","nonce":3504998958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6815","nonce":3504998958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6813","nonce":3504998958}]},"public_addr":"172.21.15.19:6809/3504998958","cluster_addr":"172.21.15.19:6811/3504998958","heartbeat_back_addr":"172.21.15.19:6815/3504998958","heartbeat_front_addr":"172.21.15.19:6813/3504998958","state":["exists","up"]},{"osd":9,"uuid":"ecc63a20-91fd-491b-ae24-1987cb89b79e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6825","nonce":659409981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6827","nonce":659409981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6831","nonce":659409981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6829","nonce":659409981}]},"public_addr":"172.21.15.19:6825/659409981","cluster_addr":"172.21.15.19:6827/659409981","heartbeat_back_addr":"172.21.15.19:6831/659409981","heartbeat_front_addr":"172.21.15.19:6829/659409981","state":["exists","up"]},{"osd":10,"uuid":"dab2c556-8182-40c4-8905-ed4d3f3eef0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6816","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6817","nonce":3054655181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6819","nonce":3054655181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6823","nonce":3054655181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6820","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6821","nonce":3054655181}]},"public_addr":"172.21.15.18:6817/3054655181","cluster_addr":"172.21.15.18:6819/3054655181","heartbeat_back_addr":"172.21.15.18:6823/3054655181","heartbeat_front_addr":"172.21.15.18:6821/3054655181","state":["exists","up"]},{"osd":11,"uuid":"6ad55028-2074-457d-9de8-83e2194b4ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6817","nonce":3207016817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6819","nonce":3207016817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6823","nonce":3207016817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6821","nonce":3207016817}]},"public_addr":"172.21.15.19:6817/3207016817","cluster_addr":"172.21.15.19:6819/3207016817","heartbeat_back_addr":"172.21.15.19:6823/3207016817","heartbeat_front_addr":"172.21.15.19:6821/3207016817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:27:49.829069+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:27:50.502721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.459409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.385240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.522717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:07.668464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:11.401425+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.247625+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.184580+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:15.874632+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.205918+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:15.833881+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2163995605":"2022-03-07T00:26:25.217327+0000","172.21.15.59:0/2251141036":"2022-03-07T00:26:25.217327+0000","172.21.15.59:6801/2847001190":"2022-03-07T00:26:25.217327+0000","172.21.15.59:6800/2847001190":"2022-03-07T00:26:25.217327+0000","172.21.15.59:0/1767239452":"2022-03-07T00:26:25.217327+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-06T00:28:30.427 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-06T00:28:30.428 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd dump --format=json 2022-03-06T00:28:30.971 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:30.972 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":30,"fsid":"509894aa-97af-4ae5-a89a-865199c2e73d","created":"2022-03-06T00:23:53.351658+0000","modified":"2022-03-06T00:28:19.475543+0000","last_up_change":"2022-03-06T00:28:17.921325+0000","last_in_change":"2022-03-06T00:27:42.604573+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-06T00:28:13.063439+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"37072002-518c-46e6-a446-34d8b1623080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6801","nonce":3083061169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6803","nonce":3083061169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6807","nonce":3083061169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":3083061169},{"type":"v1","addr":"172.21.15.18:6805","nonce":3083061169}]},"public_addr":"172.21.15.18:6801/3083061169","cluster_addr":"172.21.15.18:6803/3083061169","heartbeat_back_addr":"172.21.15.18:6807/3083061169","heartbeat_front_addr":"172.21.15.18:6805/3083061169","state":["exists","up"]},{"osd":1,"uuid":"ca74c2b9-c7f6-4694-8e9a-be3abe3d6859","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6801","nonce":4012638484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6803","nonce":4012638484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6807","nonce":4012638484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":4012638484},{"type":"v1","addr":"172.21.15.19:6805","nonce":4012638484}]},"public_addr":"172.21.15.19:6801/4012638484","cluster_addr":"172.21.15.19:6803/4012638484","heartbeat_back_addr":"172.21.15.19:6807/4012638484","heartbeat_front_addr":"172.21.15.19:6805/4012638484","state":["exists","up"]},{"osd":2,"uuid":"6fb19e44-feef-4b43-be82-0e1228ac862d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6819","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6821","nonce":1625184900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6823","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6825","nonce":1625184900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6831","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6833","nonce":1625184900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1625184900},{"type":"v1","addr":"172.21.15.59:6829","nonce":1625184900}]},"public_addr":"172.21.15.59:6821/1625184900","cluster_addr":"172.21.15.59:6825/1625184900","heartbeat_back_addr":"172.21.15.59:6833/1625184900","heartbeat_front_addr":"172.21.15.59:6829/1625184900","state":["exists","up"]},{"osd":3,"uuid":"5cb8fc2b-399c-4f4d-8468-413d140ddab8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6811","nonce":3422330840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6813","nonce":3422330840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6817","nonce":3422330840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3422330840},{"type":"v1","addr":"172.21.15.59:6815","nonce":3422330840}]},"public_addr":"172.21.15.59:6811/3422330840","cluster_addr":"172.21.15.59:6813/3422330840","heartbeat_back_addr":"172.21.15.59:6817/3422330840","heartbeat_front_addr":"172.21.15.59:6815/3422330840","state":["exists","up"]},{"osd":4,"uuid":"cd401e96-7f2b-443e-b113-8827d4b0081f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6820","nonce":3444139581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6824","nonce":3444139581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6832","nonce":3444139581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6827","nonce":3444139581},{"type":"v1","addr":"172.21.15.59:6828","nonce":3444139581}]},"public_addr":"172.21.15.59:6820/3444139581","cluster_addr":"172.21.15.59:6824/3444139581","heartbeat_back_addr":"172.21.15.59:6832/3444139581","heartbeat_front_addr":"172.21.15.59:6828/3444139581","state":["exists","up"]},{"osd":5,"uuid":"2ca11032-ca69-405a-a237-88a110504873","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6809","nonce":2660541088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6811","nonce":2660541088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6815","nonce":2660541088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6812","nonce":2660541088},{"type":"v1","addr":"172.21.15.18:6813","nonce":2660541088}]},"public_addr":"172.21.15.18:6809/2660541088","cluster_addr":"172.21.15.18:6811/2660541088","heartbeat_back_addr":"172.21.15.18:6815/2660541088","heartbeat_front_addr":"172.21.15.18:6813/2660541088","state":["exists","up"]},{"osd":6,"uuid":"364afe94-5e85-4316-b8ec-470c170ce7ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6803","nonce":516578741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6805","nonce":516578741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6809","nonce":516578741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":516578741},{"type":"v1","addr":"172.21.15.59:6807","nonce":516578741}]},"public_addr":"172.21.15.59:6803/516578741","cluster_addr":"172.21.15.59:6805/516578741","heartbeat_back_addr":"172.21.15.59:6809/516578741","heartbeat_front_addr":"172.21.15.59:6807/516578741","state":["exists","up"]},{"osd":7,"uuid":"ff8b1be4-3cef-4410-8c31-c5e815c52acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6824","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6825","nonce":2724201665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6826","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6827","nonce":2724201665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6830","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6831","nonce":2724201665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6828","nonce":2724201665},{"type":"v1","addr":"172.21.15.18:6829","nonce":2724201665}]},"public_addr":"172.21.15.18:6825/2724201665","cluster_addr":"172.21.15.18:6827/2724201665","heartbeat_back_addr":"172.21.15.18:6831/2724201665","heartbeat_front_addr":"172.21.15.18:6829/2724201665","state":["exists","up"]},{"osd":8,"uuid":"2c2c6946-fab1-47d5-a8ed-c7b76b22ea3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6809","nonce":3504998958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6811","nonce":3504998958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6815","nonce":3504998958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3504998958},{"type":"v1","addr":"172.21.15.19:6813","nonce":3504998958}]},"public_addr":"172.21.15.19:6809/3504998958","cluster_addr":"172.21.15.19:6811/3504998958","heartbeat_back_addr":"172.21.15.19:6815/3504998958","heartbeat_front_addr":"172.21.15.19:6813/3504998958","state":["exists","up"]},{"osd":9,"uuid":"ecc63a20-91fd-491b-ae24-1987cb89b79e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6825","nonce":659409981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6827","nonce":659409981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6831","nonce":659409981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":659409981},{"type":"v1","addr":"172.21.15.19:6829","nonce":659409981}]},"public_addr":"172.21.15.19:6825/659409981","cluster_addr":"172.21.15.19:6827/659409981","heartbeat_back_addr":"172.21.15.19:6831/659409981","heartbeat_front_addr":"172.21.15.19:6829/659409981","state":["exists","up"]},{"osd":10,"uuid":"dab2c556-8182-40c4-8905-ed4d3f3eef0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6816","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6817","nonce":3054655181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6819","nonce":3054655181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6823","nonce":3054655181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6820","nonce":3054655181},{"type":"v1","addr":"172.21.15.18:6821","nonce":3054655181}]},"public_addr":"172.21.15.18:6817/3054655181","cluster_addr":"172.21.15.18:6819/3054655181","heartbeat_back_addr":"172.21.15.18:6823/3054655181","heartbeat_front_addr":"172.21.15.18:6821/3054655181","state":["exists","up"]},{"osd":11,"uuid":"6ad55028-2074-457d-9de8-83e2194b4ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6817","nonce":3207016817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6819","nonce":3207016817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6823","nonce":3207016817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":3207016817},{"type":"v1","addr":"172.21.15.19:6821","nonce":3207016817}]},"public_addr":"172.21.15.19:6817/3207016817","cluster_addr":"172.21.15.19:6819/3207016817","heartbeat_back_addr":"172.21.15.19:6823/3207016817","heartbeat_front_addr":"172.21.15.19:6821/3207016817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:27:49.829069+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:27:50.502721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.459409+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.385240+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:13.522717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:07.668464+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:11.401425+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.247625+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.184580+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:15.874632+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:14.205918+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:28:15.833881+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/2163995605":"2022-03-07T00:26:25.217327+0000","172.21.15.59:0/2251141036":"2022-03-07T00:26:25.217327+0000","172.21.15.59:6801/2847001190":"2022-03-07T00:26:25.217327+0000","172.21.15.59:6800/2847001190":"2022-03-07T00:26:25.217327+0000","172.21.15.59:0/1767239452":"2022-03-07T00:26:25.217327+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-06T00:28:30.989 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.0 flush_pg_stats 2022-03-06T00:28:30.990 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.1 flush_pg_stats 2022-03-06T00:28:30.991 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.2 flush_pg_stats 2022-03-06T00:28:30.992 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.3 flush_pg_stats 2022-03-06T00:28:30.992 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.4 flush_pg_stats 2022-03-06T00:28:30.993 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.5 flush_pg_stats 2022-03-06T00:28:30.993 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.6 flush_pg_stats 2022-03-06T00:28:30.994 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.7 flush_pg_stats 2022-03-06T00:28:30.995 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.8 flush_pg_stats 2022-03-06T00:28:30.995 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.9 flush_pg_stats 2022-03-06T00:28:30.996 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.10 flush_pg_stats 2022-03-06T00:28:30.997 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph tell osd.11 flush_pg_stats 2022-03-06T00:28:31.522 INFO:teuthology.orchestra.run.smithi018.stdout:115964116997 2022-03-06T00:28:31.523 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.4 2022-03-06T00:28:31.833 INFO:teuthology.orchestra.run.smithi018.stdout:98784247814 2022-03-06T00:28:31.835 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.5 2022-03-06T00:28:31.911 INFO:teuthology.orchestra.run.smithi018.stdout:115964116997 2022-03-06T00:28:31.912 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.2 2022-03-06T00:28:31.920 INFO:teuthology.orchestra.run.smithi018.stdout:107374182406 2022-03-06T00:28:31.920 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.6 2022-03-06T00:28:31.921 INFO:teuthology.orchestra.run.smithi018.stdout:115964116997 2022-03-06T00:28:31.922 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.8 2022-03-06T00:28:31.924 INFO:teuthology.orchestra.run.smithi018.stdout:124554051590 2022-03-06T00:28:31.924 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.9 2022-03-06T00:28:31.925 INFO:teuthology.orchestra.run.smithi018.stdout:81604378636 2022-03-06T00:28:31.926 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.0 2022-03-06T00:28:31.927 INFO:teuthology.orchestra.run.smithi018.stdout:111669149703 2022-03-06T00:28:31.928 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.3 2022-03-06T00:28:32.015 INFO:teuthology.orchestra.run.smithi018.stdout:115964116999 2022-03-06T00:28:32.015 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.7 2022-03-06T00:28:32.016 INFO:teuthology.orchestra.run.smithi018.stdout:115964116997 2022-03-06T00:28:32.017 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.10 2022-03-06T00:28:32.019 INFO:teuthology.orchestra.run.smithi018.stdout:77309411339 2022-03-06T00:28:32.020 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.1 2022-03-06T00:28:32.029 INFO:teuthology.orchestra.run.smithi018.stdout:124554051589 2022-03-06T00:28:32.029 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.11 2022-03-06T00:28:32.371 INFO:teuthology.orchestra.run.smithi018.stdout:115964116996 2022-03-06T00:28:32.518 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116996 for osd.4 2022-03-06T00:28:32.529 INFO:teuthology.orchestra.run.smithi018.stdout:98784247813 2022-03-06T00:28:32.727 INFO:teuthology.orchestra.run.smithi018.stdout:115964116996 2022-03-06T00:28:32.782 INFO:teuthology.orchestra.run.smithi018.stdout:115964116996 2022-03-06T00:28:32.815 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247814 got 98784247813 for osd.5 2022-03-06T00:28:32.841 INFO:teuthology.orchestra.run.smithi018.stdout:107374182405 2022-03-06T00:28:32.845 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116996 for osd.2 2022-03-06T00:28:32.846 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116996 for osd.8 2022-03-06T00:28:32.886 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182406 got 107374182405 for osd.6 2022-03-06T00:28:32.928 INFO:teuthology.orchestra.run.smithi018.stdout:124554051589 2022-03-06T00:28:32.929 INFO:teuthology.orchestra.run.smithi018.stdout:81604378635 2022-03-06T00:28:32.945 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051590 got 124554051589 for osd.9 2022-03-06T00:28:32.955 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378636 got 81604378635 for osd.0 2022-03-06T00:28:32.982 INFO:teuthology.orchestra.run.smithi018.stdout:111669149701 2022-03-06T00:28:33.005 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149701 for osd.3 2022-03-06T00:28:33.083 INFO:teuthology.orchestra.run.smithi018.stdout:115964116995 2022-03-06T00:28:33.089 INFO:teuthology.orchestra.run.smithi018.stdout:124554051588 2022-03-06T00:28:33.103 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116995 for osd.10 2022-03-06T00:28:33.111 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051589 got 124554051588 for osd.11 2022-03-06T00:28:33.154 INFO:teuthology.orchestra.run.smithi018.stdout:115964116997 2022-03-06T00:28:33.158 INFO:teuthology.orchestra.run.smithi018.stdout:77309411337 2022-03-06T00:28:33.177 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116999 got 115964116997 for osd.7 2022-03-06T00:28:33.179 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411339 got 77309411337 for osd.1 2022-03-06T00:28:33.520 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.4 2022-03-06T00:28:33.817 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.5 2022-03-06T00:28:33.846 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.2 2022-03-06T00:28:33.847 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.8 2022-03-06T00:28:33.887 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.6 2022-03-06T00:28:33.946 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.9 2022-03-06T00:28:33.956 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.0 2022-03-06T00:28:34.006 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.3 2022-03-06T00:28:34.097 INFO:teuthology.orchestra.run.smithi018.stdout:115964116998 2022-03-06T00:28:34.104 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.10 2022-03-06T00:28:34.113 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.11 2022-03-06T00:28:34.163 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116998 for osd.4 2022-03-06T00:28:34.164 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:34.180 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.7 2022-03-06T00:28:34.181 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph osd last-stat-seq osd.1 2022-03-06T00:28:34.679 INFO:teuthology.orchestra.run.smithi018.stdout:115964116998 2022-03-06T00:28:34.745 INFO:teuthology.orchestra.run.smithi018.stdout:124554051590 2022-03-06T00:28:34.761 INFO:teuthology.orchestra.run.smithi018.stdout:98784247815 2022-03-06T00:28:34.821 INFO:teuthology.orchestra.run.smithi018.stdout:115964116998 2022-03-06T00:28:34.839 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116998 for osd.2 2022-03-06T00:28:34.840 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:34.842 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051590 got 124554051590 for osd.9 2022-03-06T00:28:34.842 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:34.845 INFO:tasks.rook.ceph_manager.ceph:need seq 98784247814 got 98784247815 for osd.5 2022-03-06T00:28:34.845 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:34.851 INFO:teuthology.orchestra.run.smithi018.stdout:81604378636 2022-03-06T00:28:34.885 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116998 for osd.8 2022-03-06T00:28:34.885 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:34.888 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378636 got 81604378636 for osd.0 2022-03-06T00:28:34.888 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:34.896 INFO:teuthology.orchestra.run.smithi018.stdout:107374182406 2022-03-06T00:28:34.917 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182406 got 107374182406 for osd.6 2022-03-06T00:28:34.918 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:35.007 INFO:teuthology.orchestra.run.smithi018.stdout:115964116997 2022-03-06T00:28:35.020 INFO:teuthology.orchestra.run.smithi018.stdout:111669149703 2022-03-06T00:28:35.031 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116997 got 115964116997 for osd.10 2022-03-06T00:28:35.031 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:35.037 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149703 got 111669149703 for osd.3 2022-03-06T00:28:35.037 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:35.071 INFO:teuthology.orchestra.run.smithi018.stdout:124554051589 2022-03-06T00:28:35.089 INFO:tasks.rook.ceph_manager.ceph:need seq 124554051589 got 124554051589 for osd.11 2022-03-06T00:28:35.089 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:35.134 INFO:teuthology.orchestra.run.smithi018.stdout:115964116999 2022-03-06T00:28:35.153 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116999 got 115964116999 for osd.7 2022-03-06T00:28:35.153 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:35.217 INFO:teuthology.orchestra.run.smithi018.stdout:77309411339 2022-03-06T00:28:35.884 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411339 got 77309411339 for osd.1 2022-03-06T00:28:35.885 DEBUG:teuthology.parallel:result is None 2022-03-06T00:28:35.885 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-06T00:28:35.886 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph pg dump --format=json 2022-03-06T00:28:36.437 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:36.441 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2022-03-06T00:28:36.467 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2022-03-06T00:28:34.536609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":64888,"kb_used_data":4824,"kb_used_omap":0,"kb_used_meta":59584,"kb_avail":1124876936,"statfs":{"total":1151940427776,"available":1151873982464,"internally_reserved":0,"allocated":4939776,"data_stored":2893632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61014016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003540"},"pg_stats":[{"pgid":"1.0","version":"30'137","reported_seq":184,"reported_epoch":30,"state":"active+clean","last_fresh":"2022-03-06T00:28:29.316373+0000","last_change":"2022-03-06T00:28:17.206131+0000","last_active":"2022-03-06T00:28:29.316373+0000","last_peered":"2022-03-06T00:28:29.316373+0000","last_clean":"2022-03-06T00:28:29.316373+0000","last_became_active":"2022-03-06T00:28:17.204411+0000","last_became_peered":"2022-03-06T00:28:17.204411+0000","last_unstale":"2022-03-06T00:28:29.316373+0000","last_undegraded":"2022-03-06T00:28:29.316373+0000","last_fullsized":"2022-03-06T00:28:29.316373+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-06T00:28:13.539014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-06T00:28:13.539014+0000","last_clean_scrub_stamp":"2022-03-06T00:28:13.539014+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-07T03:25:22.874626+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10],"acting":[7,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":11,"up_from":29,"seq":124554051590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.363}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.98}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]}]},{"osd":9,"up_from":29,"seq":124554051591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9590000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.651}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]}]},{"osd":10,"up_from":27,"seq":115964116997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":811008,"data_stored":637832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3450000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.958}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4489999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0710000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.54}]}]},{"osd":8,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3919999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.958}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0590000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.044}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0760000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]}]},{"osd":7,"up_from":27,"seq":115964116999,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":811008,"data_stored":637832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.012}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.262}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0680000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3479999999999999}]}]},{"osd":4,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.890999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0859999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1120000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8779999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2629999999999999}]}]},{"osd":0,"up_from":19,"seq":81604378637,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740008,"statfs":{"total":95995035648,"available":95989768192,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9820000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.992}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8560000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.516}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7729999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9339999999999999}]}]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740008,"statfs":{"total":95995035648,"available":95989768192,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]}]},{"osd":5,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740008,"statfs":{"total":95995035648,"available":95989768192,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6870000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2149999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]}]},{"osd":6,"up_from":25,"seq":107374182406,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740008,"statfs":{"total":95995035648,"available":95989768192,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8560000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4529999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2850000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.351}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5990000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8360000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.117}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0670000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.04}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]}]},{"osd":2,"up_from":27,"seq":115964116998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":811008,"data_stored":637832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.667999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8660000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.377000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.571999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-06T00:28:36.469 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph pg dump --format=json 2022-03-06T00:28:36.981 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:36.983 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2022-03-06T00:28:37.008 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2022-03-06T00:28:36.537228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":64888,"kb_used_data":4824,"kb_used_omap":0,"kb_used_meta":59584,"kb_avail":1124876936,"statfs":{"total":1151940427776,"available":1151873982464,"internally_reserved":0,"allocated":4939776,"data_stored":2893632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61014016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003624"},"pg_stats":[{"pgid":"1.0","version":"30'137","reported_seq":184,"reported_epoch":30,"state":"active+clean","last_fresh":"2022-03-06T00:28:29.316373+0000","last_change":"2022-03-06T00:28:17.206131+0000","last_active":"2022-03-06T00:28:29.316373+0000","last_peered":"2022-03-06T00:28:29.316373+0000","last_clean":"2022-03-06T00:28:29.316373+0000","last_became_active":"2022-03-06T00:28:17.204411+0000","last_became_peered":"2022-03-06T00:28:17.204411+0000","last_unstale":"2022-03-06T00:28:29.316373+0000","last_undegraded":"2022-03-06T00:28:29.316373+0000","last_fullsized":"2022-03-06T00:28:29.316373+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-06T00:28:13.539014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-06T00:28:13.539014+0000","last_clean_scrub_stamp":"2022-03-06T00:28:13.539014+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-07T03:25:22.874626+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10],"acting":[7,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":11,"up_from":29,"seq":124554051590,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.363}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.98}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.161}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3879999999999999}]}]},{"osd":9,"up_from":29,"seq":124554051591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9590000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.651}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]}]},{"osd":10,"up_from":27,"seq":115964116997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":811008,"data_stored":637832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3450000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.958}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4489999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0710000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.54}]}]},{"osd":8,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3919999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.958}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0590000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.044}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0760000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]}]},{"osd":7,"up_from":27,"seq":115964116999,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":811008,"data_stored":637832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8169999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.012}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.262}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0680000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3479999999999999}]}]},{"osd":4,"up_from":27,"seq":115964116998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.890999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0859999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1120000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8779999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2629999999999999}]}]},{"osd":0,"up_from":19,"seq":81604378637,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740008,"statfs":{"total":95995035648,"available":95989768192,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9820000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.992}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8560000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.516}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7729999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9339999999999999}]}]},{"osd":1,"up_from":18,"seq":77309411340,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740008,"statfs":{"total":95995035648,"available":95989768192,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0129999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6639999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.129}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2879999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2130000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0609999999999999}]}]},{"osd":5,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740008,"statfs":{"total":95995035648,"available":95989768192,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.76}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6870000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2149999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6240000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]}]},{"osd":6,"up_from":25,"seq":107374182407,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740008,"statfs":{"total":95995035648,"available":95989768192,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.966}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8560000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4529999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5150000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2850000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.351}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5990000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5080,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740072,"statfs":{"total":95995035648,"available":95989833728,"internally_reserved":0,"allocated":278528,"data_stored":108904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4379999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0009999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.113}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3690000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3290000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1440000000000001}]}]},{"osd":2,"up_from":27,"seq":115964116998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":792,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":811008,"data_stored":637832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.667999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8660000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.377000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.571999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-06T00:28:37.011 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-06T00:28:37.012 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-06T00:28:37.013 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-06T00:28:37.013 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph health --format=json 2022-03-06T00:28:37.569 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:37.570 INFO:teuthology.orchestra.run.smithi018.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-06T00:28:37.590 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-06T00:28:37.591 INFO:tasks.rook:Rook complete, yielding 2022-03-06T00:28:37.591 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-06T00:28:37.604 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph -s 2022-03-06T00:28:38.253 INFO:teuthology.orchestra.run.smithi018.stdout: cluster: 2022-03-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi018.stdout: id: 509894aa-97af-4ae5-a89a-865199c2e73d 2022-03-06T00:28:38.254 INFO:teuthology.orchestra.run.smithi018.stdout: health: HEALTH_OK 2022-03-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi018.stdout: services: 2022-03-06T00:28:38.255 INFO:teuthology.orchestra.run.smithi018.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi018.stdout: mgr: a(active, since 2m) 2022-03-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi018.stdout: osd: 12 osds: 12 up (since 20s), 12 in (since 55s) 2022-03-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:38.256 INFO:teuthology.orchestra.run.smithi018.stdout: data: 2022-03-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi018.stdout: pools: 1 pools, 1 pgs 2022-03-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi018.stdout: objects: 2 objects, 577 KiB 2022-03-06T00:28:38.257 INFO:teuthology.orchestra.run.smithi018.stdout: usage: 63 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi018.stdout: pgs: 1 active+clean 2022-03-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:38.258 INFO:teuthology.orchestra.run.smithi018.stdout: io: 2022-03-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi018.stdout: client: 767 B/s rd, 22 KiB/s wr, 0 op/s rd, 1 op/s wr 2022-03-06T00:28:38.259 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:28:38.280 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status 2022-03-06T00:28:38.997 INFO:teuthology.orchestra.run.smithi018.stdout:Backend: rook 2022-03-06T00:28:38.997 INFO:teuthology.orchestra.run.smithi018.stdout:Available: Yes 2022-03-06T00:28:39.027 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ps 2022-03-06T00:28:39.608 INFO:teuthology.orchestra.run.smithi018.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-06T00:28:39.609 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi018 running (2m) 0s ago 2m - - 02136b44099a 2022-03-06T00:28:39.610 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi019 running (2m) 0s ago 2m - - 02136b44099a 2022-03-06T00:28:39.610 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi059 running (62s) 0s ago 62s - - 02136b44099a 2022-03-06T00:28:39.610 INFO:teuthology.orchestra.run.smithi018.stdout:mgr.a smithi059 running (2m) 0s ago 2m - - 02136b44099a 2022-03-06T00:28:39.611 INFO:teuthology.orchestra.run.smithi018.stdout:mon.a smithi019 running (4m) 0s ago 5m - - 02136b44099a 2022-03-06T00:28:39.611 INFO:teuthology.orchestra.run.smithi018.stdout:mon.b smithi018 running (4m) 0s ago 4m - - 02136b44099a 2022-03-06T00:28:39.612 INFO:teuthology.orchestra.run.smithi018.stdout:mon.c smithi059 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:28:39.612 INFO:teuthology.orchestra.run.smithi018.stdout:osd.0 smithi018 running (73s) 0s ago 73s - - 02136b44099a 2022-03-06T00:28:39.613 INFO:teuthology.orchestra.run.smithi018.stdout:osd.1 smithi019 running (70s) 0s ago 70s - - 02136b44099a 2022-03-06T00:28:39.613 INFO:teuthology.orchestra.run.smithi018.stdout:osd.10 smithi018 running (48s) 0s ago 48s - - 02136b44099a 2022-03-06T00:28:39.613 INFO:teuthology.orchestra.run.smithi018.stdout:osd.11 smithi019 running (49s) 0s ago 50s - - 02136b44099a 2022-03-06T00:28:39.614 INFO:teuthology.orchestra.run.smithi018.stdout:osd.2 smithi059 running (63s) 0s ago 63s - - 02136b44099a 2022-03-06T00:28:39.614 INFO:teuthology.orchestra.run.smithi018.stdout:osd.3 smithi059 running (62s) 0s ago 62s - - 02136b44099a 2022-03-06T00:28:39.615 INFO:teuthology.orchestra.run.smithi018.stdout:osd.4 smithi059 running (60s) 0s ago 61s - - 02136b44099a 2022-03-06T00:28:39.615 INFO:teuthology.orchestra.run.smithi018.stdout:osd.5 smithi018 running (59s) 0s ago 59s - - 02136b44099a 2022-03-06T00:28:39.616 INFO:teuthology.orchestra.run.smithi018.stdout:osd.6 smithi059 running (55s) 0s ago 56s - - 02136b44099a 2022-03-06T00:28:39.616 INFO:teuthology.orchestra.run.smithi018.stdout:osd.7 smithi018 running (53s) 0s ago 53s - - 02136b44099a 2022-03-06T00:28:39.617 INFO:teuthology.orchestra.run.smithi018.stdout:osd.8 smithi019 running (53s) 0s ago 53s - - 02136b44099a 2022-03-06T00:28:39.617 INFO:teuthology.orchestra.run.smithi018.stdout:osd.9 smithi019 running (51s) 0s ago 52s - - 02136b44099a 2022-03-06T00:28:39.632 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls 2022-03-06T00:28:40.385 INFO:teuthology.orchestra.run.smithi018.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-06T00:28:40.386 INFO:teuthology.orchestra.run.smithi018.stdout:crash 3/3 0s ago 2m * 2022-03-06T00:28:40.386 INFO:teuthology.orchestra.run.smithi018.stdout:mgr 1/1 0s ago 2m count:1 2022-03-06T00:28:40.387 INFO:teuthology.orchestra.run.smithi018.stdout:mon 3/3 0s ago 5m count:3 2022-03-06T00:28:40.387 INFO:teuthology.orchestra.run.smithi018.stdout:osd 24 0s ago 74s 2022-03-06T00:28:40.390 INFO:teuthology.orchestra.run.smithi018.stdout:osd.all-available-devices 0 0s ago - * 2022-03-06T00:28:40.420 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch device ls 2022-03-06T00:28:40.994 INFO:teuthology.orchestra.run.smithi018.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-06T00:28:40.994 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:28:40.994 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:28:40.994 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:28:40.995 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:28:40.995 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:28:40.995 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:28:40.996 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:28:40.996 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:28:40.996 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:28:40.996 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:28:40.996 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:28:40.997 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:28:41.018 INFO:teuthology.run_tasks:Running task exec... 2022-03-06T00:28:41.029 INFO:teuthology.task.exec:Executing custom commands... 2022-03-06T00:28:41.030 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi018.front.sepia.ceph.com 2022-03-06T00:28:41.031 DEBUG:teuthology.orchestra.run.smithi018:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-06T00:28:41.031 DEBUG:teuthology.orchestra.run.smithi018:> toolbox() { 2022-03-06T00:28:41.032 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-06T00:28:41.032 DEBUG:teuthology.orchestra.run.smithi018:> } 2022-03-06T00:28:41.032 DEBUG:teuthology.orchestra.run.smithi018:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-06T00:28:41.032 DEBUG:teuthology.orchestra.run.smithi018:> toolbox ceph orch osd rm 0 --force 2022-03-06T00:28:41.033 DEBUG:teuthology.orchestra.run.smithi018:> removed_pv="" 2022-03-06T00:28:41.033 DEBUG:teuthology.orchestra.run.smithi018:> while [ "$removed_pv" = "" ] 2022-03-06T00:28:41.033 DEBUG:teuthology.orchestra.run.smithi018:> do 2022-03-06T00:28:41.034 DEBUG:teuthology.orchestra.run.smithi018:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-06T00:28:41.034 DEBUG:teuthology.orchestra.run.smithi018:> sleep 3s 2022-03-06T00:28:41.034 DEBUG:teuthology.orchestra.run.smithi018:> done 2022-03-06T00:28:41.035 DEBUG:teuthology.orchestra.run.smithi018:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-06T00:28:41.035 DEBUG:teuthology.orchestra.run.smithi018:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-06T00:28:41.036 DEBUG:teuthology.orchestra.run.smithi018:> toolbox ceph orch device zap $host $target_path --force 2022-03-06T00:28:41.036 DEBUG:teuthology.orchestra.run.smithi018:> zap_completion="0" 2022-03-06T00:28:41.037 DEBUG:teuthology.orchestra.run.smithi018:> while [ "$zap_completion" = "0" ] 2022-03-06T00:28:41.037 DEBUG:teuthology.orchestra.run.smithi018:> do 2022-03-06T00:28:41.037 DEBUG:teuthology.orchestra.run.smithi018:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-06T00:28:41.038 DEBUG:teuthology.orchestra.run.smithi018:> sleep 3s 2022-03-06T00:28:41.038 DEBUG:teuthology.orchestra.run.smithi018:> done 2022-03-06T00:28:41.038 DEBUG:teuthology.orchestra.run.smithi018:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-06T00:28:41.039 DEBUG:teuthology.orchestra.run.smithi018:> toolbox ceph orch apply osd --all-available-devices 2022-03-06T00:28:41.039 DEBUG:teuthology.orchestra.run.smithi018:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-06T00:28:41.039 DEBUG:teuthology.orchestra.run.smithi018:> num_osd="0" 2022-03-06T00:28:41.040 DEBUG:teuthology.orchestra.run.smithi018:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-06T00:28:41.041 DEBUG:teuthology.orchestra.run.smithi018:> do 2022-03-06T00:28:41.042 DEBUG:teuthology.orchestra.run.smithi018:> echo "waiting for osd to come back up" 2022-03-06T00:28:41.042 DEBUG:teuthology.orchestra.run.smithi018:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-06T00:28:41.042 DEBUG:teuthology.orchestra.run.smithi018:> sleep 30s 2022-03-06T00:28:41.043 DEBUG:teuthology.orchestra.run.smithi018:> done 2022-03-06T00:28:41.043 DEBUG:teuthology.orchestra.run.smithi018:> ' 2022-03-06T00:28:41.057 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-06T00:28:41.057 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:28:41.058 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-06T00:28:43.259 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:28:43.774 INFO:teuthology.orchestra.run.smithi018.stderr:+ orig_num_osd=12 2022-03-06T00:28:43.775 INFO:teuthology.orchestra.run.smithi018.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-06T00:28:43.775 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-06T00:28:43.860 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:28:44.533 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.520000 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-06T00:28:44.534 INFO:tasks.rook.operator.smithi018.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", AllowUnsupported: true}, 2022-03-06T00:28:44.534 INFO:tasks.rook.operator.smithi018.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-06T00:28:44.534 INFO:tasks.rook.operator.smithi018.stdout:   ... // 3 identical fields 2022-03-06T00:28:44.534 INFO:tasks.rook.operator.smithi018.stdout:   Config: nil, 2022-03-06T00:28:44.535 INFO:tasks.rook.operator.smithi018.stdout:   Selection: {}, 2022-03-06T00:28:44.535 INFO:tasks.rook.operator.smithi018.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-06T00:28:44.535 INFO:tasks.rook.operator.smithi018.stdout:   {Name: "smithi018-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi018-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:28:44.535 INFO:tasks.rook.operator.smithi018.stdout:-  { 2022-03-06T00:28:44.536 INFO:tasks.rook.operator.smithi018.stdout:-  Name: "smithi018-lv-2", 2022-03-06T00:28:44.536 INFO:tasks.rook.operator.smithi018.stdout:-  Count: 1, 2022-03-06T00:28:44.536 INFO:tasks.rook.operator.smithi018.stdout:-  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-06T00:28:44.536 INFO:tasks.rook.operator.smithi018.stdout:-  }, 2022-03-06T00:28:44.537 INFO:tasks.rook.operator.smithi018.stdout:   {Name: "smithi018-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi018-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:28:44.537 INFO:tasks.rook.operator.smithi018.stdout:   {Name: "smithi018-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi018-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:28:44.537 INFO:tasks.rook.operator.smithi018.stdout:   ... // 8 identical elements 2022-03-06T00:28:44.537 INFO:tasks.rook.operator.smithi018.stdout:   }, 2022-03-06T00:28:44.538 INFO:tasks.rook.operator.smithi018.stdout:   }, 2022-03-06T00:28:44.538 INFO:tasks.rook.operator.smithi018.stdout:   Annotations: nil, 2022-03-06T00:28:44.538 INFO:tasks.rook.operator.smithi018.stdout:   Labels: nil, 2022-03-06T00:28:44.539 INFO:tasks.rook.operator.smithi018.stdout:   ... // 20 identical fields 2022-03-06T00:28:44.540 INFO:tasks.rook.operator.smithi018.stdout:  } 2022-03-06T00:28:44.540 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.520128 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-06T00:28:44.541 INFO:tasks.rook.operator.smithi018.stdout:I0306 00:28:44.520285 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-06T00:28:44.541 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.520610 I | operator: successfully started the controller-runtime manager 2022-03-06T00:28:44.556 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.550035 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:28:44.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.550066 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:28:44.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.550091 I | operator: setting up schemes 2022-03-06T00:28:44.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.552263 I | operator: setting up the controller-runtime manager 2022-03-06T00:28:44.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.855443 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:28:44.885 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.877584 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:28:44.885 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.877675 I | ceph-cluster-controller: successfully started 2022-03-06T00:28:44.885 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.877774 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:28:44.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.877828 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:28:44.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.877873 I | ceph-block-pool-controller: successfully started 2022-03-06T00:28:44.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.877913 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:28:44.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.878119 I | ceph-object-realm-controller: successfully started 2022-03-06T00:28:44.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.878247 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:28:44.887 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.878347 I | ceph-object-zone-controller: successfully started 2022-03-06T00:28:44.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.878632 I | ceph-object-controller: successfully started 2022-03-06T00:28:44.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.878765 I | ceph-file-controller: successfully started 2022-03-06T00:28:44.890 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.878848 I | ceph-nfs-controller: successfully started 2022-03-06T00:28:44.891 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.878903 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:28:44.891 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.878945 I | ceph-client-controller: successfully started 2022-03-06T00:28:44.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.879156 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:28:44.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.879392 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:28:44.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.879494 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:28:44.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.879561 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:28:44.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.879640 I | ceph-bucket-topic: successfully started 2022-03-06T00:28:44.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.879674 I | ceph-bucket-notification: successfully started 2022-03-06T00:28:44.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.879696 I | ceph-bucket-notification: successfully started 2022-03-06T00:28:44.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.879724 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:28:44.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.881150 I | operator: starting the controller-runtime manager 2022-03-06T00:28:44.990 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.983799 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:28:44.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.983853 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:28:44.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.983864 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:28:44.998 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:44.992301 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:28:45.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.021456 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:28:45.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.053762 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:28:45.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.053841 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-06T00:28:45.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.053855 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:28:45.065 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.055469 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-06T00:28:45.065 INFO:tasks.rook.operator.smithi018.stdout:I0306 00:28:45.055595 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:28:45.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.078409 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:28:45.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.342207 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-06T00:28:45.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.398600 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:28:45.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.850028 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-06T00:28:45.867 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858117 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-06T00:28:45.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858152 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-06T00:28:45.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858158 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-06T00:28:45.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858164 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-06T00:28:45.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858169 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-06T00:28:45.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858175 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-06T00:28:45.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858183 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-06T00:28:45.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858189 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-06T00:28:45.871 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858194 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-06T00:28:45.871 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858199 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-06T00:28:45.871 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858204 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-06T00:28:45.871 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858210 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-06T00:28:45.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858214 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-06T00:28:45.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858218 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-06T00:28:45.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858224 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-06T00:28:45.873 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858285 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:28:45.873 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:45.858334 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:28:48.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:48.445412 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-06T00:28:48.598 INFO:teuthology.orchestra.run.smithi018.stdout:Success 2022-03-06T00:28:48.626 INFO:teuthology.orchestra.run.smithi018.stderr:+ removed_pv= 2022-03-06T00:28:48.627 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' '' = '' ']' 2022-03-06T00:28:48.627 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl get pv 2022-03-06T00:28:48.627 INFO:teuthology.orchestra.run.smithi018.stderr:++ grep Released 2022-03-06T00:28:48.628 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f1 -d ' ' 2022-03-06T00:28:48.751 INFO:teuthology.orchestra.run.smithi018.stderr:+ removed_pv=smithi018-lv-2 2022-03-06T00:28:48.751 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 3s 2022-03-06T00:28:51.748 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' smithi018-lv-2 = '' ']' 2022-03-06T00:28:51.749 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl get pv smithi018-lv-2 -o 'jsonpath={.spec.local.path}' 2022-03-06T00:28:51.868 INFO:teuthology.orchestra.run.smithi018.stderr:+ target_path=/dev/vg_nvme/lv_2 2022-03-06T00:28:51.868 INFO:teuthology.orchestra.run.smithi018.stderr:++ echo smithi018-lv-2 2022-03-06T00:28:51.869 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f1 -d - 2022-03-06T00:28:51.869 INFO:teuthology.orchestra.run.smithi018.stderr:+ host=smithi018 2022-03-06T00:28:51.870 INFO:teuthology.orchestra.run.smithi018.stderr:+ toolbox ceph orch device zap smithi018 /dev/vg_nvme/lv_2 --force 2022-03-06T00:28:51.870 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi018 /dev/vg_nvme/lv_2 --force 2022-03-06T00:28:51.931 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:28:52.516 INFO:teuthology.orchestra.run.smithi018.stdout:/dev/vg_nvme/lv_2 on smithi018 zapped 2022-03-06T00:28:52.543 INFO:teuthology.orchestra.run.smithi018.stderr:+ zap_completion=0 2022-03-06T00:28:52.544 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 0 = 0 ']' 2022-03-06T00:28:52.544 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-06T00:28:52.719 INFO:teuthology.orchestra.run.smithi018.stderr:+ zap_completion= 2022-03-06T00:28:52.719 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 3s 2022-03-06T00:28:53.384 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376300 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-06T00:28:53.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376342 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-06T00:28:53.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376350 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:28:53.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376354 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:28:53.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376359 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:28:53.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376363 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:28:53.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376368 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:28:53.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376372 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:28:53.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376383 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:28:53.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376387 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:28:53.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376391 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:28:53.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376395 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:28:53.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376399 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:28:53.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376403 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:28:53.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376408 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-06T00:28:53.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376413 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-06T00:28:53.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376417 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-06T00:28:53.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376427 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-06T00:28:53.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376432 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-06T00:28:53.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376436 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:28:53.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376441 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:28:53.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376446 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-06T00:28:53.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376450 I | ceph-csi: Kubernetes version is 1.23 2022-03-06T00:28:53.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376456 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-06T00:28:53.395 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.376460 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-06T00:28:53.766 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.760886 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-06T00:28:53.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.767157 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:28:53.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.767180 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:28:53.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.767186 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:28:53.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.767190 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:28:53.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.767193 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:28:53.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.767197 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:28:53.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:53.767202 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-06T00:28:54.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.581988 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:28:54.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.582108 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:28:54.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.582208 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:28:54.591 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.583225 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:28:54.592 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.583248 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:28:54.592 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.583254 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:28:54.818 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.811131 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:28:54.840 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.834733 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-06T00:28:54.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.882620 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:28:54.906 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:54.899934 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:28:55.225 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:55.219266 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:28:55.225 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:55.219305 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:28:55.225 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:55.219318 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-06T00:28:55.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:55.340867 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:28:55.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:55.340883 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:28:55.347 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:55.340886 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:28:55.383 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:55.377833 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:28:55.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:55.395004 I | ceph-csi: successfully started CSI CephFS driver 2022-03-06T00:28:55.719 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' '' = 0 ']' 2022-03-06T00:28:55.720 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl patch pv smithi018-lv-2 -p '{"spec":{"claimRef": null}}' 2022-03-06T00:28:56.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.147614 I | op-mon: start running mons 2022-03-06T00:28:56.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.220863 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:28:56.227 INFO:teuthology.orchestra.run.smithi018.stdout:persistentvolume/smithi018-lv-2 patched 2022-03-06T00:28:56.230 INFO:teuthology.orchestra.run.smithi018.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-06T00:28:56.230 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-06T00:28:56.457 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.451164 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:28:56.461 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.456225 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-06T00:28:56.461 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.456265 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:28:56.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.461633 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-06T00:28:56.470 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:28:56.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.480540 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.19:6789","172.21.15.18:6789","172.21.15.59:6789"]}] data:a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:2] 2022-03-06T00:28:56.652 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.646484 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:28:56.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:56.646760 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:28:57.713 INFO:teuthology.orchestra.run.smithi018.stdout:['Success'] 2022-03-06T00:28:57.715 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.709982 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{ 2022-03-06T00:28:57.715 INFO:tasks.rook.operator.smithi018.stdout:   CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", AllowUnsupported: true}, 2022-03-06T00:28:57.716 INFO:tasks.rook.operator.smithi018.stdout:   Storage: v1.StorageScopeSpec{ 2022-03-06T00:28:57.716 INFO:tasks.rook.operator.smithi018.stdout:   ... // 3 identical fields 2022-03-06T00:28:57.716 INFO:tasks.rook.operator.smithi018.stdout:   Config: nil, 2022-03-06T00:28:57.718 INFO:tasks.rook.operator.smithi018.stdout:   Selection: {}, 2022-03-06T00:28:57.718 INFO:tasks.rook.operator.smithi018.stdout:   StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-06T00:28:57.718 INFO:tasks.rook.operator.smithi018.stdout:   ... // 9 identical elements 2022-03-06T00:28:57.719 INFO:tasks.rook.operator.smithi018.stdout:   {Name: "smithi059-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi059-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:28:57.719 INFO:tasks.rook.operator.smithi018.stdout:   {Name: "smithi059-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi059-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:28:57.719 INFO:tasks.rook.operator.smithi018.stdout:+  { 2022-03-06T00:28:57.720 INFO:tasks.rook.operator.smithi018.stdout:+  Name: "smithi018-lv-2", 2022-03-06T00:28:57.720 INFO:tasks.rook.operator.smithi018.stdout:+  Count: 1, 2022-03-06T00:28:57.720 INFO:tasks.rook.operator.smithi018.stdout:+  VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-06T00:28:57.720 INFO:tasks.rook.operator.smithi018.stdout:+  }, 2022-03-06T00:28:57.720 INFO:tasks.rook.operator.smithi018.stdout:   }, 2022-03-06T00:28:57.721 INFO:tasks.rook.operator.smithi018.stdout:   }, 2022-03-06T00:28:57.721 INFO:tasks.rook.operator.smithi018.stdout:   Annotations: nil, 2022-03-06T00:28:57.721 INFO:tasks.rook.operator.smithi018.stdout:   Labels: nil, 2022-03-06T00:28:57.722 INFO:tasks.rook.operator.smithi018.stdout:   ... // 20 identical fields 2022-03-06T00:28:57.722 INFO:tasks.rook.operator.smithi018.stdout:  } 2022-03-06T00:28:57.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.710087 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-06T00:28:57.722 INFO:tasks.rook.operator.smithi018.stdout:I0306 00:28:57.710715 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-06T00:28:57.740 INFO:teuthology.orchestra.run.smithi018.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-06T00:28:57.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.767817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to initialize ceph cluster info: failed to save admin keyring secret: failed to update secret for rook-ceph-admin-keyring: context canceled 2022-03-06T00:28:57.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.768095 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:28:57.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.768270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to load cluster info: failed to get mon secrets: context canceled 2022-03-06T00:28:57.779 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.768318 I | operator: successfully started the controller-runtime manager 2022-03-06T00:28:57.856 INFO:teuthology.orchestra.run.smithi018.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-06T00:28:57.857 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.851516 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:28:57.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.851545 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:28:57.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.851573 I | operator: setting up schemes 2022-03-06T00:28:57.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:57.853851 I | operator: setting up the controller-runtime manager 2022-03-06T00:28:57.865 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=0 2022-03-06T00:28:57.868 INFO:teuthology.orchestra.run.smithi018.stdout:waiting for osd to come back up 2022-03-06T00:28:57.869 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 0 '!=' 12 ']' 2022-03-06T00:28:57.870 INFO:teuthology.orchestra.run.smithi018.stderr:+ echo 'waiting for osd to come back up' 2022-03-06T00:28:57.870 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-06T00:28:57.870 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-06T00:28:57.870 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:28:57.948 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:28:58.164 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.158406 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:28:58.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.160917 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:28:58.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.160956 I | ceph-cluster-controller: successfully started 2022-03-06T00:28:58.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161012 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:28:58.170 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161038 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:28:58.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161062 I | ceph-block-pool-controller: successfully started 2022-03-06T00:28:58.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161108 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:28:58.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161131 I | ceph-object-realm-controller: successfully started 2022-03-06T00:28:58.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161145 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:28:58.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161159 I | ceph-object-zone-controller: successfully started 2022-03-06T00:28:58.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161303 I | ceph-object-controller: successfully started 2022-03-06T00:28:58.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161343 I | ceph-file-controller: successfully started 2022-03-06T00:28:58.172 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161374 I | ceph-nfs-controller: successfully started 2022-03-06T00:28:58.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161407 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:28:58.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161436 I | ceph-client-controller: successfully started 2022-03-06T00:28:58.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161456 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:28:58.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161486 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:28:58.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161508 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:28:58.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161529 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:28:58.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161550 I | ceph-bucket-topic: successfully started 2022-03-06T00:28:58.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161566 I | ceph-bucket-notification: successfully started 2022-03-06T00:28:58.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161581 I | ceph-bucket-notification: successfully started 2022-03-06T00:28:58.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.161594 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:28:58.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.162384 I | operator: starting the controller-runtime manager 2022-03-06T00:28:58.273 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.264820 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:28:58.279 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.264869 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:28:58.280 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.264886 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:28:58.289 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.268221 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:28:58.290 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.269987 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:28:58.502 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=11 2022-03-06T00:28:58.502 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 30s 2022-03-06T00:28:58.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.872581 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:28:58.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.872629 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-06T00:28:58.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.872636 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:28:58.885 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:58.876912 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-06T00:28:58.886 INFO:tasks.rook.operator.smithi018.stdout:I0306 00:28:58.877120 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:28:59.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.057836 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:28:59.218 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.212595 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-06T00:28:59.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.488348 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:28:59.624 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618079 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-06T00:28:59.624 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618103 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-06T00:28:59.625 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618109 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-06T00:28:59.625 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618116 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-06T00:28:59.625 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618124 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-06T00:28:59.626 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618130 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-06T00:28:59.626 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618136 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-06T00:28:59.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618141 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-06T00:28:59.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618146 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-06T00:28:59.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618152 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-06T00:28:59.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618157 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-06T00:28:59.628 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618164 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-06T00:28:59.628 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618170 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-06T00:28:59.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618175 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-06T00:28:59.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618181 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-06T00:28:59.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618201 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-06T00:28:59.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618286 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:28:59.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:28:59.618308 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:29:06.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.238692 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:29:07.764 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.238730 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:29:07.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.238747 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:29:07.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.257706 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:29:07.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.329898 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:29:07.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.330095 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:29:07.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672132 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-06T00:29:07.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672196 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-06T00:29:07.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672208 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:29:07.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672217 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:29:07.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672224 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:29:07.771 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672230 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:29:07.771 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672236 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:29:07.771 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672242 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:29:07.771 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672249 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:29:07.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672254 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:29:07.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672261 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:29:07.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672268 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:29:07.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672275 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:29:07.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672281 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:29:07.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672287 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-06T00:29:07.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672293 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-06T00:29:07.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672300 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-06T00:29:07.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672306 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-06T00:29:07.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672312 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-06T00:29:07.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672319 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:29:07.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672326 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:29:07.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672333 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-06T00:29:07.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672339 I | ceph-csi: Kubernetes version is 1.23 2022-03-06T00:29:07.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672347 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-06T00:29:07.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.672353 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-06T00:29:07.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:06.789898 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:29:07.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.053423 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-06T00:29:07.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.060302 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:29:07.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.060336 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:29:07.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.060340 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:29:07.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.060342 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:29:07.777 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.060345 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:29:07.777 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.060348 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:29:07.777 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.060353 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-06T00:29:07.777 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.486811 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:29:07.777 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.486842 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:29:07.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.486856 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:29:07.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.503917 I | op-mon: start running mons 2022-03-06T00:29:07.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.577896 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:29:07.778 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.664000 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-06T00:29:07.845 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:07.840553 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.19:6789","172.21.15.18:6789","172.21.15.59:6789"]}] data:a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:2] 2022-03-06T00:29:08.047 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.043091 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:29:08.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.043129 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:29:08.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.043141 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-06T00:29:08.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.125746 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:29:08.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.125793 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:29:08.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.125809 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:29:08.202 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.197280 I | ceph-csi: successfully started CSI CephFS driver 2022-03-06T00:29:08.277 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.271195 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:29:08.278 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:08.271430 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:29:09.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:09.648119 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:29:09.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:09.654672 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-06T00:29:09.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:09.654697 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:29:09.664 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:09.659744 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-06T00:29:10.449 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:10.444608 I | op-mon: targeting the mon count 3 2022-03-06T00:29:10.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:10.452604 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:29:10.819 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:10.815264 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:29:10.820 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:10.815312 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:29:11.177 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:11.172537 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:29:11.177 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:11.172572 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:29:11.531 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:11.526731 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:29:11.532 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:11.526763 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:29:11.993 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:11.989033 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:29:11.994 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:11.989062 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:29:12.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.357638 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:29:12.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.357673 I | op-mon: checking for basic quorum with existing mons 2022-03-06T00:29:12.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.357684 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:29:12.365 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.357695 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:29:12.365 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.357702 I | op-mon: setting mon endpoints for hostnetwork mode 2022-03-06T00:29:12.380 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.375874 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["172.21.15.19:6789","172.21.15.18:6789","172.21.15.59:6789"]}] data:a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 mapping:{"node":{"a":{"Name":"smithi019","Hostname":"smithi019","Address":"172.21.15.19"},"b":{"Name":"smithi018","Hostname":"smithi018","Address":"172.21.15.18"},"c":{"Name":"smithi059","Hostname":"smithi059","Address":"172.21.15.59"}}} maxMonId:2] 2022-03-06T00:29:12.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.382544 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:29:12.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.382662 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:29:12.434 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.428697 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-06T00:29:12.447 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.442983 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-06T00:29:12.447 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.443004 I | op-mon: waiting for mon quorum with [b c a] 2022-03-06T00:29:12.654 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:12.649969 I | op-mon: mons running: [b c a] 2022-03-06T00:29:13.060 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.056261 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:29:13.088 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.084056 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-06T00:29:13.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.095478 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-06T00:29:13.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.095512 I | op-mon: waiting for mon quorum with [b c a] 2022-03-06T00:29:13.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.254034 I | op-mon: mons running: [b c a] 2022-03-06T00:29:13.675 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.671105 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:29:13.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.676990 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:29:13.690 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.686627 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:29:13.691 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.686650 I | op-mon: waiting for mon quorum with [b c a] 2022-03-06T00:29:13.856 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:13.852048 I | op-mon: mons running: [b c a] 2022-03-06T00:29:14.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:14.267221 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:29:14.272 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:14.267249 I | op-mon: mons created: 3 2022-03-06T00:29:14.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:14.714777 I | op-mon: waiting for mon quorum with [b c a] 2022-03-06T00:29:14.750 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:14.744692 I | op-mon: mons running: [b c a] 2022-03-06T00:29:15.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:15.170980 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:29:15.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:15.171014 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-06T00:29:15.605 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:15.601171 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-06T00:29:16.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:16.027968 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-06T00:29:16.454 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:16.449747 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-06T00:29:16.933 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:16.928645 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-06T00:29:16.933 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:16.928670 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-06T00:29:17.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:17.389460 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-06T00:29:17.808 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:17.804019 I | cephclient: successfully enabled msgr2 protocol 2022-03-06T00:29:17.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:17.804053 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-06T00:29:17.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:17.804060 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-06T00:29:18.242 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.237637 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-06T00:29:18.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.263660 I | op-mgr: start running mgr 2022-03-06T00:29:18.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.263696 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-06T00:29:18.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.735870 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-06T00:29:18.748 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.744189 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-06T00:29:18.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.747693 I | op-mgr: setting services to point to mgr "a" 2022-03-06T00:29:18.917 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.912783 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-06T00:29:18.918 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.912883 I | op-mgr: successful modules: balancer 2022-03-06T00:29:18.938 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.934482 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-06T00:29:18.938 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.934518 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-06T00:29:18.971 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:18.967057 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-06T00:29:19.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113160 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-1" 2022-03-06T00:29:19.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113195 I | op-osd: OSD PVC "smithi018-lv-1-data-0j686x" already exists 2022-03-06T00:29:19.124 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113210 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-3" 2022-03-06T00:29:19.124 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113222 I | op-osd: OSD PVC "smithi018-lv-3-data-0t695v" already exists 2022-03-06T00:29:19.125 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113231 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi018-lv-4" 2022-03-06T00:29:19.125 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113243 I | op-osd: OSD PVC "smithi018-lv-4-data-0qhqwb" already exists 2022-03-06T00:29:19.126 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113253 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi019-lv-1" 2022-03-06T00:29:19.126 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113266 I | op-osd: OSD PVC "smithi019-lv-1-data-0n4sw2" already exists 2022-03-06T00:29:19.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113275 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi019-lv-2" 2022-03-06T00:29:19.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113286 I | op-osd: OSD PVC "smithi019-lv-2-data-0dgfhd" already exists 2022-03-06T00:29:19.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113296 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi019-lv-3" 2022-03-06T00:29:19.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113307 I | op-osd: OSD PVC "smithi019-lv-3-data-08q647" already exists 2022-03-06T00:29:19.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113316 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi019-lv-4" 2022-03-06T00:29:19.128 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113326 I | op-osd: OSD PVC "smithi019-lv-4-data-0t7rhl" already exists 2022-03-06T00:29:19.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113335 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi059-lv-1" 2022-03-06T00:29:19.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113345 I | op-osd: OSD PVC "smithi059-lv-1-data-0js9sd" already exists 2022-03-06T00:29:19.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113355 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi059-lv-2" 2022-03-06T00:29:19.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113365 I | op-osd: OSD PVC "smithi059-lv-2-data-0knbvw" already exists 2022-03-06T00:29:19.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113377 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi059-lv-3" 2022-03-06T00:29:19.132 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113389 I | op-osd: OSD PVC "smithi059-lv-3-data-0fp8vd" already exists 2022-03-06T00:29:19.132 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113399 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi059-lv-4" 2022-03-06T00:29:19.132 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113411 I | op-osd: OSD PVC "smithi059-lv-4-data-0jq895" already exists 2022-03-06T00:29:19.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.113420 I | op-osd: creating 1 new PVCs for device set "smithi018-lv-2" 2022-03-06T00:29:19.355 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.351135 I | op-osd: successfully provisioned PVC "smithi018-lv-2-data-0mfzn4" 2022-03-06T00:29:19.384 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377107 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-0j686x" 2022-03-06T00:29:19.384 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377127 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-1-data-0j686x" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.384 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377131 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0t695v" 2022-03-06T00:29:19.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377134 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-3-data-0t695v" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377137 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0qhqwb" 2022-03-06T00:29:19.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377140 I | op-osd: skipping OSD prepare job creation for PVC "smithi018-lv-4-data-0qhqwb" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377145 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-1-data-0n4sw2" 2022-03-06T00:29:19.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377148 I | op-osd: skipping OSD prepare job creation for PVC "smithi019-lv-1-data-0n4sw2" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377152 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-2-data-0dgfhd" 2022-03-06T00:29:19.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377155 I | op-osd: skipping OSD prepare job creation for PVC "smithi019-lv-2-data-0dgfhd" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377158 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-3-data-08q647" 2022-03-06T00:29:19.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377162 I | op-osd: skipping OSD prepare job creation for PVC "smithi019-lv-3-data-08q647" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377165 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-4-data-0t7rhl" 2022-03-06T00:29:19.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377170 I | op-osd: skipping OSD prepare job creation for PVC "smithi019-lv-4-data-0t7rhl" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377183 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-1-data-0js9sd" 2022-03-06T00:29:19.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377189 I | op-osd: skipping OSD prepare job creation for PVC "smithi059-lv-1-data-0js9sd" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377192 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-2-data-0knbvw" 2022-03-06T00:29:19.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377195 I | op-osd: skipping OSD prepare job creation for PVC "smithi059-lv-2-data-0knbvw" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377199 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-3-data-0fp8vd" 2022-03-06T00:29:19.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377203 I | op-osd: skipping OSD prepare job creation for PVC "smithi059-lv-3-data-0fp8vd" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377206 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-4-data-0jq895" 2022-03-06T00:29:19.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377210 I | op-osd: skipping OSD prepare job creation for PVC "smithi059-lv-4-data-0jq895" because OSD daemon using the PVC already exists 2022-03-06T00:29:19.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.377213 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0mfzn4" 2022-03-06T00:29:19.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.732057 I | op-osd: started OSD provisioning job for PVC "smithi018-lv-2-data-0mfzn4" 2022-03-06T00:29:19.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.732085 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-06T00:29:19.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.732090 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-06T00:29:19.918 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:19.914475 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0mfzn4 is "starting" 2022-03-06T00:29:20.021 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.016181 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:20.027 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.023425 I | op-osd: updating OSD 1 on PVC "smithi019-lv-1-data-0n4sw2" 2022-03-06T00:29:20.027 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.023466 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-1-data-0n4sw2" 2022-03-06T00:29:20.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.183064 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:20.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.219158 I | op-osd: updating OSD 10 on PVC "smithi018-lv-4-data-0qhqwb" 2022-03-06T00:29:20.224 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.219301 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-4-data-0qhqwb" 2022-03-06T00:29:20.597 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.592069 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:20.635 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.631533 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-06T00:29:20.639 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.634739 I | op-mgr: successful modules: dashboard 2022-03-06T00:29:20.639 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.635150 I | op-mgr: successful modules: prometheus 2022-03-06T00:29:20.664 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.660234 I | op-osd: updating OSD 11 on PVC "smithi019-lv-3-data-08q647" 2022-03-06T00:29:20.665 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.660267 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-3-data-08q647" 2022-03-06T00:29:20.918 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.913946 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:20.924 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.920861 I | op-osd: updating OSD 2 on PVC "smithi059-lv-1-data-0js9sd" 2022-03-06T00:29:20.924 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:20.920898 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-1-data-0js9sd" 2022-03-06T00:29:21.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:21.327584 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:21.428 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:21.423458 I | op-osd: updating OSD 3 on PVC "smithi059-lv-2-data-0knbvw" 2022-03-06T00:29:21.428 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:21.423495 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-2-data-0knbvw" 2022-03-06T00:29:22.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:22.127646 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:22.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:22.222976 I | op-osd: updating OSD 4 on PVC "smithi059-lv-3-data-0fp8vd" 2022-03-06T00:29:22.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:22.223011 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-3-data-0fp8vd" 2022-03-06T00:29:22.362 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:22.357954 I | op-mgr: successful modules: orchestrator modules 2022-03-06T00:29:22.933 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:22.929089 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:23.026 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:23.022067 I | op-osd: updating OSD 5 on PVC "smithi018-lv-1-data-0j686x" 2022-03-06T00:29:23.026 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:23.022101 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-1-data-0j686x" 2022-03-06T00:29:23.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:23.731161 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:23.828 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:23.823622 I | op-osd: updating OSD 6 on PVC "smithi059-lv-4-data-0jq895" 2022-03-06T00:29:23.828 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:23.823668 I | op-osd: OSD will have its main bluestore block on "smithi059-lv-4-data-0jq895" 2022-03-06T00:29:24.532 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:24.528385 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:24.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:24.624392 I | op-osd: updating OSD 7 on PVC "smithi018-lv-3-data-0t695v" 2022-03-06T00:29:24.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:24.624427 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-3-data-0t695v" 2022-03-06T00:29:25.330 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:25.326824 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:25.427 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:25.423652 I | op-osd: updating OSD 8 on PVC "smithi019-lv-4-data-0t7rhl" 2022-03-06T00:29:25.428 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:25.423681 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-4-data-0t7rhl" 2022-03-06T00:29:26.134 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:26.130554 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:29:26.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:26.223016 I | op-osd: updating OSD 9 on PVC "smithi019-lv-2-data-0dgfhd" 2022-03-06T00:29:26.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:26.223053 I | op-osd: OSD will have its main bluestore block on "smithi019-lv-2-data-0dgfhd" 2022-03-06T00:29:28.502 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 11 '!=' 12 ']' 2022-03-06T00:29:29.243 INFO:teuthology.orchestra.run.smithi018.stderr:+ echo 'waiting for osd to come back up' 2022-03-06T00:29:29.244 INFO:teuthology.orchestra.run.smithi018.stdout:waiting for osd to come back up 2022-03-06T00:29:29.245 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-06T00:29:29.245 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:29:29.246 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-06T00:29:29.246 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:29:29.246 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=11 2022-03-06T00:29:29.246 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 30s 2022-03-06T00:29:42.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:42.476206 I | op-osd: OSD orchestration status for node smithi018-lv-2-data-0mfzn4 is "orchestrating" 2022-03-06T00:29:42.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:42.919081 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0mfzn4 is "orchestrating" 2022-03-06T00:29:48.949 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:48.944692 I | op-osd: OSD orchestration status for PVC smithi018-lv-2-data-0mfzn4 is "completed" 2022-03-06T00:29:48.949 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:48.944718 I | op-osd: creating OSD 0 on PVC "smithi018-lv-2-data-0mfzn4" 2022-03-06T00:29:48.949 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:48.944728 I | op-osd: OSD will have its main bluestore block on "smithi018-lv-2-data-0mfzn4" 2022-03-06T00:29:50.144 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:50.140551 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-06T00:29:50.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:29:50.140586 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:29:59.105 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 11 '!=' 12 ']' 2022-03-06T00:29:59.108 INFO:teuthology.orchestra.run.smithi018.stderr:+ echo 'waiting for osd to come back up' 2022-03-06T00:29:59.109 INFO:teuthology.orchestra.run.smithi018.stdout:waiting for osd to come back up 2022-03-06T00:29:59.111 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-06T00:29:59.111 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:29:59.112 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-06T00:29:59.176 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:29:59.838 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=11 2022-03-06T00:29:59.839 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 30s 2022-03-06T00:30:29.843 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 11 '!=' 12 ']' 2022-03-06T00:30:29.844 INFO:teuthology.orchestra.run.smithi018.stdout:waiting for osd to come back up 2022-03-06T00:30:29.845 INFO:teuthology.orchestra.run.smithi018.stderr:+ echo 'waiting for osd to come back up' 2022-03-06T00:30:29.845 INFO:teuthology.orchestra.run.smithi018.stderr:++ toolbox ceph osd stat 2022-03-06T00:30:29.845 INFO:teuthology.orchestra.run.smithi018.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:30:29.846 INFO:teuthology.orchestra.run.smithi018.stderr:++ cut -f3 -d ' ' 2022-03-06T00:30:29.910 INFO:teuthology.orchestra.run.smithi018.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:30:30.397 INFO:teuthology.orchestra.run.smithi018.stderr:+ num_osd=12 2022-03-06T00:30:30.398 INFO:teuthology.orchestra.run.smithi018.stderr:+ sleep 30s 2022-03-06T00:31:00.419 INFO:teuthology.orchestra.run.smithi018.stderr:+ '[' 12 '!=' 12 ']' 2022-03-06T00:31:00.421 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-06T00:31:00.434 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch status 2022-03-06T00:31:01.249 INFO:teuthology.orchestra.run.smithi018.stdout:Backend: rook 2022-03-06T00:31:01.250 INFO:teuthology.orchestra.run.smithi018.stdout:Available: Yes 2022-03-06T00:31:01.276 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ps 2022-03-06T00:31:01.874 INFO:teuthology.orchestra.run.smithi018.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-06T00:31:01.875 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi018 running (5m) 0s ago 5m - - 02136b44099a 2022-03-06T00:31:01.875 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi019 running (4m) 0s ago 4m - - 02136b44099a 2022-03-06T00:31:01.875 INFO:teuthology.orchestra.run.smithi018.stdout:crashcollector.crash smithi059 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.876 INFO:teuthology.orchestra.run.smithi018.stdout:mgr.a smithi059 running (5m) 0s ago 5m - - 02136b44099a 2022-03-06T00:31:01.876 INFO:teuthology.orchestra.run.smithi018.stdout:mon.a smithi019 running (7m) 0s ago 7m - - 02136b44099a 2022-03-06T00:31:01.876 INFO:teuthology.orchestra.run.smithi018.stdout:mon.b smithi018 running (7m) 0s ago 7m - - 02136b44099a 2022-03-06T00:31:01.876 INFO:teuthology.orchestra.run.smithi018.stdout:mon.c smithi059 running (5m) 0s ago 6m - - 02136b44099a 2022-03-06T00:31:01.876 INFO:teuthology.orchestra.run.smithi018.stdout:osd.0 smithi018 running (72s) 0s ago 72s - - 02136b44099a 2022-03-06T00:31:01.877 INFO:teuthology.orchestra.run.smithi018.stdout:osd.1 smithi019 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.877 INFO:teuthology.orchestra.run.smithi018.stdout:osd.10 smithi018 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.877 INFO:teuthology.orchestra.run.smithi018.stdout:osd.11 smithi019 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.877 INFO:teuthology.orchestra.run.smithi018.stdout:osd.2 smithi059 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.878 INFO:teuthology.orchestra.run.smithi018.stdout:osd.3 smithi059 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.878 INFO:teuthology.orchestra.run.smithi018.stdout:osd.4 smithi059 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.878 INFO:teuthology.orchestra.run.smithi018.stdout:osd.5 smithi018 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.878 INFO:teuthology.orchestra.run.smithi018.stdout:osd.6 smithi059 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.878 INFO:teuthology.orchestra.run.smithi018.stdout:osd.7 smithi018 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.879 INFO:teuthology.orchestra.run.smithi018.stdout:osd.8 smithi019 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.879 INFO:teuthology.orchestra.run.smithi018.stdout:osd.9 smithi019 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:31:01.893 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls 2022-03-06T00:31:02.586 INFO:teuthology.orchestra.run.smithi018.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-06T00:31:02.586 INFO:teuthology.orchestra.run.smithi018.stdout:crash 3/3 0s ago 5m * 2022-03-06T00:31:02.587 INFO:teuthology.orchestra.run.smithi018.stdout:mgr 1/1 0s ago 5m count:1 2022-03-06T00:31:02.587 INFO:teuthology.orchestra.run.smithi018.stdout:mon 3/3 0s ago 7m count:3 2022-03-06T00:31:02.587 INFO:teuthology.orchestra.run.smithi018.stdout:osd 24 0s ago 3m 2022-03-06T00:31:02.587 INFO:teuthology.orchestra.run.smithi018.stdout:osd.all-available-devices 0 0s ago - * 2022-03-06T00:31:02.612 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch host ls 2022-03-06T00:31:03.186 INFO:teuthology.orchestra.run.smithi018.stdout:HOST ADDR LABELS STATUS 2022-03-06T00:31:03.187 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 172.21.15.18/smithi018 2022-03-06T00:31:03.187 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 172.21.15.19/smithi019 2022-03-06T00:31:03.187 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 172.21.15.59/smithi059 2022-03-06T00:31:03.187 INFO:teuthology.orchestra.run.smithi018.stdout:3 hosts in cluster 2022-03-06T00:31:03.205 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch device ls 2022-03-06T00:31:03.775 INFO:teuthology.orchestra.run.smithi018.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-06T00:31:03.775 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:31:03.775 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:31:03.776 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:31:03.776 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:31:03.776 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:31:03.776 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:31:03.777 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:31:03.777 INFO:teuthology.orchestra.run.smithi018.stdout:smithi019 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:31:03.777 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:31:03.777 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:31:03.778 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:31:03.778 INFO:teuthology.orchestra.run.smithi018.stdout:smithi059 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:31:03.792 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch apply rgw foo 2022-03-06T00:31:04.360 INFO:teuthology.orchestra.run.smithi018.stdout:Created 2022-03-06T00:31:04.361 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:04.356570 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-06T00:31:04.382 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch apply mds foo 2022-03-06T00:31:04.524 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:04.522106 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:04.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:04.522180 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:31:05.010 INFO:teuthology.orchestra.run.smithi018.stdout:Created 2022-03-06T00:31:05.011 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:05.009991 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-06T00:31:05.032 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch apply rbd-mirror 2022-03-06T00:31:05.578 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:05.565208 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:05.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:05.565263 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:31:06.256 INFO:teuthology.orchestra.run.smithi018.stdout:Success 2022-03-06T00:31:06.278 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch apply nfs foo --port 12777 2022-03-06T00:31:06.293 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:06.291047 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:06.293 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:06.291119 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:31:09.961 INFO:teuthology.orchestra.run.smithi018.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-06T00:31:09.961 INFO:teuthology.orchestra.run.smithi018.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-06T00:31:09.989 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-06T00:31:10.000 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:31:11.977 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:11.974429 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:31:12.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:12.845597 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:31:12.981 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:12.979683 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:31:13.347 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:13.345259 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-06T00:31:13.347 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:13.345294 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-06T00:31:13.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:13.345310 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-06T00:31:13.821 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:13.820033 I | ceph-object-controller: reconciling object store deployments 2022-03-06T00:31:14.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:14.005421 I | ceph-object-controller: ceph object store gateway service running at 10.108.241.30 2022-03-06T00:31:14.008 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:14.005466 I | ceph-object-controller: reconciling object store pools 2022-03-06T00:31:14.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:14.257500 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-06T00:31:14.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:14.759812 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-06T00:31:14.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:14.781614 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-06T00:31:15.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:15.334965 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-06T00:31:15.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:15.828425 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi019": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:31:16.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:16.860369 I | op-mds: setting mds config flags 2022-03-06T00:31:16.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:16.860397 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-06T00:31:17.030 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:31:17.030 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:58.000000Z", "last_refresh": "2022-03-06T00:31:16.831395Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:16.831395Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:31:16.831395Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:31:16.831395Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:31:16.831395Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:16.831395Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-06T00:31:16.831395Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:16.831395Z", "running": 0, "size": 1}}] 2022-03-06T00:31:17.085 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch rm mds.foo 2022-03-06T00:31:17.741 INFO:teuthology.orchestra.run.smithi018.stdout:Removed cephfilesystems/foo 2022-03-06T00:31:17.763 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch rm rbd-mirror 2022-03-06T00:31:18.178 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:18.174448 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-06T00:31:18.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:18.184640 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-06T00:31:18.408 INFO:teuthology.orchestra.run.smithi018.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-06T00:31:18.415 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch rm rgw.foo 2022-03-06T00:31:18.544 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:18.541231 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-06T00:31:19.610 INFO:teuthology.orchestra.run.smithi018.stdout:Removed cephobjectstores/foo 2022-03-06T00:31:19.633 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:31:20.450 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:31:20.450 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:58.000000Z", "last_refresh": "2022-03-06T00:31:20.229905Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:20.229905Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:31:20.229905Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:31:20.229905Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:31:20.229905Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:20.229905Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:20.229905Z", "running": 0, "size": 1}}] 2022-03-06T00:31:21.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:21.658527 I | op-mds: setting mds config flags 2022-03-06T00:31:21.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:21.658561 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-06T00:31:24.249 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:24.247423 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-06T00:31:25.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:25.388165 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-06T00:31:25.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:25.388406 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-06T00:31:27.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:27.485078 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-06T00:31:27.492 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:27.488227 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-06T00:31:27.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:27.490443 I | ceph-file-controller: creating filesystem "foo" 2022-03-06T00:31:27.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:27.490770 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-06T00:31:30.001 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:29.999690 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-06T00:31:30.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:30.026960 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:30.497 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:31:32.159 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:31:32.159 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:58.000000Z", "last_refresh": "2022-03-06T00:31:31.287896Z", "running": 6, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:31.287896Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:31:31.287896Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:31:31.287896Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:31:31.287896Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:31.287896Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:31.287896Z", "running": 0, "size": 1}}] 2022-03-06T00:31:34.097 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:34.095005 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:34.086+0000 7f68c7e5a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:34.097 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:34.086+0000 7f68c7e5a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:34.097 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:34.086+0000 7f68c7e5a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:34.098 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:34.110 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:34.108449 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:34.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:34.797713 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-06T00:31:34.926 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:34.923897 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:34.914+0000 7f17f110a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:34.926 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:34.914+0000 7f17f110a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:34.926 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:34.914+0000 7f17f110a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:34.927 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:34.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:34.941579 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:35.597 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:35.595548 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:35.586+0000 7f6466103380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:35.598 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:35.586+0000 7f6466103380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:35.598 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:35.586+0000 7f6466103380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:35.598 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:35.610 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:35.608123 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:36.422 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:36.416274 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:36.406+0000 7f73e4831380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:36.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:36.406+0000 7f73e4831380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:36.429 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:36.406+0000 7f73e4831380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:36.430 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:36.457 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:36.445603 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:37.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:37.405879 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:37.398+0000 7f72e062f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:37.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:37.398+0000 7f72e062f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:37.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:37.398+0000 7f72e062f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:37.409 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:37.417 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:37.416312 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:38.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:38.031792 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:38.022+0000 7f98a37d6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:38.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:38.022+0000 7f98a37d6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:38.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:38.022+0000 7f98a37d6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:38.037 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:38.070 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:38.069002 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:38.721 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:38.719542 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:38.710+0000 7f733303e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:38.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:38.710+0000 7f733303e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:38.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:38.710+0000 7f733303e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:38.722 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:38.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:38.753775 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:39.418 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:39.416758 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-06T00:31:39.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:39.453948 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:39.442+0000 7fbfb39da380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:39.459 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:39.442+0000 7fbfb39da380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:39.459 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:39.442+0000 7fbfb39da380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:39.460 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:39.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:39.485944 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:40.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:40.117093 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:40.110+0000 7f6f7db33380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:40.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:40.110+0000 7f6f7db33380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:40.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:40.110+0000 7f6f7db33380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:40.121 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:40.204 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:40.201520 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-06T00:31:40.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:40.741949 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:41.436 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:41.434230 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:41.426+0000 7fc6f985f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:41.436 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:41.426+0000 7fc6f985f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:41.437 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:41.426+0000 7fc6f985f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:41.437 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:42.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:42.000514 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-06T00:31:42.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:42.000542 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-06T00:31:42.180 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:31:42.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:42.734643 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:42.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:42.737637 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-06T00:31:42.952 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:31:42.952 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:31:42.713599Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:42.713599Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:31:42.713599Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:31:42.713599Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:31:42.713599Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:42.713599Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:42.713599Z", "running": 0, "size": 1}}] 2022-03-06T00:31:43.551 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:43.549161 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-06T00:31:44.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:44.046461 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-06T00:31:44.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:44.046485 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-06T00:31:44.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:44.046491 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-06T00:31:44.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:44.640101 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-06T00:31:45.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:44.998139 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-06T00:31:45.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:45.675897 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-06T00:31:45.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:45.675921 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-06T00:31:45.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:45.675927 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-06T00:31:46.375 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:46.373008 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-06T00:31:46.375 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:46.373042 I | ceph-file-controller: downing filesystem "foo" 2022-03-06T00:31:46.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:46.587667 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:47.513 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:47.510947 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:47.502+0000 7f37626a3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:47.514 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:47.502+0000 7f37626a3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:47.514 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:47.502+0000 7f37626a3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:47.514 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:48.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:48.615856 I | ceph-file-controller: downed filesystem "foo" 2022-03-06T00:31:52.619 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:52.616576 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-06T00:31:52.619 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:52.616599 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-06T00:31:52.978 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:31:53.765 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:31:53.766 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:31:53.518743Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:53.518743Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:31:53.518743Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:31:53.518743Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:31:53.518743Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:53.518743Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:53.518743Z", "running": 0, "size": 1}}] 2022-03-06T00:31:55.180 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:55.178149 I | cephclient: purge completed for pool "foo-metadata" 2022-03-06T00:31:55.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:55.577112 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-06T00:31:55.580 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:55.577176 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-06T00:31:57.522 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:57.520298 I | cephclient: purge completed for pool "foo-data0" 2022-03-06T00:31:57.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:57.520405 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-06T00:31:57.594 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:57.592324 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-06T00:31:57.600 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:57.598189 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-06T00:31:57.604 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:57.599493 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-06T00:31:57.604 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:57.599633 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-06T00:31:57.823 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:57.820973 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:31:58.544 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:58.538938 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:58.529+0000 7f0c560ea380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:58.544 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:58.529+0000 7f0c560ea380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:58.544 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:31:58.529+0000 7f0c560ea380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:58.545 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:59.267 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:59.264734 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi018": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:31:59.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:31:59.520404 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi018": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:32:01.160 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:32:01.158144 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi019": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:32:03.799 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:32:04.518 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:32:04.519 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:32:04.337523Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:32:04.337523Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:32:04.337523Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:32:04.337523Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:04.337523Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:04.337523Z", "running": 0, "size": 1}}] 2022-03-06T00:32:14.545 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:32:15.284 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:32:15.285 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:32:15.090930Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:32:15.090930Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:32:15.090930Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:32:15.090930Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:15.090930Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:15.090930Z", "running": 0, "size": 1}}] 2022-03-06T00:32:19.029 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:32:19.025624 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:32:19.718 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:32:19.714825 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:32:19.688+0000 7fec2bab7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:32:19.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:32:19.688+0000 7fec2bab7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:32:19.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:32:19.688+0000 7fec2bab7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:32:19.723 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:32:25.314 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:32:26.025 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:32:26.026 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:32:25.839478Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:32:25.839478Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:32:25.839478Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:32:25.839478Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:25.839478Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:25.839478Z", "running": 0, "size": 1}}] 2022-03-06T00:32:36.053 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:32:36.765 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:32:36.766 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:32:36.579914Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:32:36.579914Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:32:36.579914Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:32:36.579914Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:36.579914Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:36.579914Z", "running": 0, "size": 1}}] 2022-03-06T00:32:46.789 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:32:47.565 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:32:47.565 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:32:47.381452Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:32:47.381452Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:32:47.381452Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:32:47.381452Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:47.381452Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:47.381452Z", "running": 0, "size": 1}}] 2022-03-06T00:32:57.591 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:32:58.546 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:32:58.547 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:32:58.339023Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:32:58.339023Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:32:58.339023Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:32:58.339023Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:58.339023Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:58.339023Z", "running": 0, "size": 1}}] 2022-03-06T00:33:00.687 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:33:00.682142 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:33:01.350 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:33:01.341001 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:33:01.331+0000 7f06311ed380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:33:01.350 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:33:01.331+0000 7f06311ed380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:33:01.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:33:01.331+0000 7f06311ed380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:33:01.351 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:33:08.571 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:33:09.314 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:33:09.315 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:33:09.137475Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:33:09.137475Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:33:09.137475Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:33:09.137475Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:09.137475Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:09.137475Z", "running": 0, "size": 1}}] 2022-03-06T00:33:19.340 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:33:20.055 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:33:20.055 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:33:19.879353Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:33:19.879353Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:33:19.879353Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:33:19.879353Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:19.879353Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:19.879353Z", "running": 0, "size": 1}}] 2022-03-06T00:33:30.087 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:33:30.859 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:33:30.859 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:33:30.641334Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:33:30.641334Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:33:30.641334Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:33:30.641334Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:30.641334Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:30.641334Z", "running": 0, "size": 1}}] 2022-03-06T00:33:40.887 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:33:41.684 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:33:41.684 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:33:41.455752Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:33:41.455752Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:33:41.455752Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:33:41.455752Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:41.455752Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:41.455752Z", "running": 0, "size": 1}}] 2022-03-06T00:33:51.710 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:33:52.448 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:33:52.448 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:33:52.243924Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:33:52.243924Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:33:52.243924Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:33:52.243924Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:52.243924Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:52.243924Z", "running": 0, "size": 1}}] 2022-03-06T00:34:02.473 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:34:03.247 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:34:03.248 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:34:03.065468Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:34:03.065468Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:34:03.065468Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:34:03.065468Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:03.065468Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:03.065468Z", "running": 0, "size": 1}}] 2022-03-06T00:34:13.275 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:34:13.972 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:34:13.972 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:34:13.798982Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:34:13.798982Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:34:13.798982Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:34:13.798982Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:13.798982Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:13.798982Z", "running": 0, "size": 1}}] 2022-03-06T00:34:23.273 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:34:23.267163 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:34:23.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:34:23.969714 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:34:23.956+0000 7f5b7ba0d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:34:23.976 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:34:23.956+0000 7f5b7ba0d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:34:23.976 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:34:23.956+0000 7f5b7ba0d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:34:23.976 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:34:23.999 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:34:24.813 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:34:24.814 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:34:24.626472Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:34:24.626472Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:34:24.626472Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:34:24.626472Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:24.626472Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:24.626472Z", "running": 0, "size": 1}}] 2022-03-06T00:34:34.845 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:34:35.547 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:34:35.548 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:34:35.375499Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:34:35.375499Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:34:35.375499Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:34:35.375499Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:35.375499Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:35.375499Z", "running": 0, "size": 1}}] 2022-03-06T00:34:45.577 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:34:46.314 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:34:46.315 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:34:46.132523Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:34:46.132523Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:34:46.132523Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:34:46.132523Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:46.132523Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:46.132523Z", "running": 0, "size": 1}}] 2022-03-06T00:34:56.346 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:34:57.131 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:34:57.132 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:34:56.920957Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:34:56.920957Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:34:56.920957Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:34:56.920957Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:56.920957Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:56.920957Z", "running": 0, "size": 1}}] 2022-03-06T00:35:07.156 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:35:07.852 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:35:07.852 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:35:07.683234Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:35:07.683234Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:35:07.683234Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:35:07.683234Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:07.683234Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:07.683234Z", "running": 0, "size": 1}}] 2022-03-06T00:35:17.877 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:35:18.639 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:35:18.640 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:35:18.455665Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:35:18.455665Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:35:18.455665Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:35:18.455665Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:18.455665Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:18.455665Z", "running": 0, "size": 1}}] 2022-03-06T00:35:28.672 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:35:29.384 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:35:29.385 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:35:29.205537Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:35:29.205537Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:35:29.205537Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:35:29.205537Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:29.205537Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:29.205537Z", "running": 0, "size": 1}}] 2022-03-06T00:35:39.406 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:35:40.159 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:35:40.160 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:35:39.977137Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:35:39.977137Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:35:39.977137Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:35:39.977137Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:39.977137Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:39.977137Z", "running": 0, "size": 1}}] 2022-03-06T00:35:50.183 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:35:50.821 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:35:50.821 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:35:50.649507Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:35:50.649507Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:35:50.649507Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:35:50.649507Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:50.649507Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:50.649507Z", "running": 0, "size": 1}}] 2022-03-06T00:36:00.850 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:36:01.574 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:36:01.575 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:36:01.397498Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:36:01.397498Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:36:01.397498Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:36:01.397498Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:01.397498Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:01.397498Z", "running": 0, "size": 1}}] 2022-03-06T00:36:11.603 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:36:12.567 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:36:12.570 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:36:12.182750Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:36:12.182750Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:36:12.182750Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:36:12.182750Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:12.182750Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:12.182750Z", "running": 0, "size": 1}}] 2022-03-06T00:36:22.590 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:36:23.344 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:36:23.345 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:36:23.148531Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:36:23.148531Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:36:23.148531Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:36:23.148531Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:23.148531Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:23.148531Z", "running": 0, "size": 1}}] 2022-03-06T00:36:33.379 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:36:34.095 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:36:34.096 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:36:33.928908Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:36:33.928908Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:36:33.928908Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:36:33.928908Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:33.928908Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:33.928908Z", "running": 0, "size": 1}}] 2022-03-06T00:36:44.123 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:36:44.886 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:36:44.886 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:36:44.697305Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:36:44.697305Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:36:44.697305Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:36:44.697305Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:44.697305Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:44.697305Z", "running": 0, "size": 1}}] 2022-03-06T00:36:54.911 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:36:55.656 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:36:55.658 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:36:55.480125Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:36:55.480125Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:36:55.480125Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:36:55.480125Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:55.480125Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:55.480125Z", "running": 0, "size": 1}}] 2022-03-06T00:37:05.683 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:37:06.543 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:37:06.544 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:37:06.361687Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:37:06.361687Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:37:06.361687Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:37:06.361687Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:06.361687Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:06.361687Z", "running": 0, "size": 1}}] 2022-03-06T00:37:07.837 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:37:07.824273 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:37:08.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:37:08.533065 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:37:08.522+0000 7f09f9e83380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:37:08.544 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:37:08.522+0000 7f09f9e83380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:37:08.544 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:37:08.522+0000 7f09f9e83380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:37:08.545 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:37:16.571 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:37:17.338 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:37:17.338 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:37:17.168725Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:37:17.168725Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:37:17.168725Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:37:17.168725Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:17.168725Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:17.168725Z", "running": 0, "size": 1}}] 2022-03-06T00:37:27.365 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:37:28.125 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:37:28.125 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:37:27.907166Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:37:27.907166Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:37:27.907166Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:37:27.907166Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:27.907166Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:27.907166Z", "running": 0, "size": 1}}] 2022-03-06T00:37:38.154 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:37:38.918 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:37:38.919 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:37:38.726810Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:37:38.726810Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:37:38.726810Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:37:38.726810Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:38.726810Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:38.726810Z", "running": 0, "size": 1}}] 2022-03-06T00:37:48.945 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:37:49.696 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:37:49.697 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:37:49.483371Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:37:49.483371Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:37:49.483371Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:37:49.483371Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:49.483371Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:49.483371Z", "running": 0, "size": 1}}] 2022-03-06T00:37:59.719 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:38:00.478 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:38:00.479 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:38:00.297928Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:38:00.297928Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:38:00.297928Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:38:00.297928Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:00.297928Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:00.297928Z", "running": 0, "size": 1}}] 2022-03-06T00:38:10.505 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:38:11.237 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:38:11.238 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:38:11.052484Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:38:11.052484Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:38:11.052484Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:38:11.052484Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:11.052484Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:11.052484Z", "running": 0, "size": 1}}] 2022-03-06T00:38:21.270 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:38:22.071 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:38:22.071 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:38:21.877101Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:38:21.877101Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:38:21.877101Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:38:21.877101Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:21.877101Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:21.877101Z", "running": 0, "size": 1}}] 2022-03-06T00:38:32.099 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:38:32.933 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:38:32.933 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:38:32.673529Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:38:32.673529Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:38:32.673529Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:38:32.673529Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:32.673529Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:32.673529Z", "running": 0, "size": 1}}] 2022-03-06T00:38:42.956 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:38:43.776 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:38:43.776 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:38:43.539272Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:38:43.539272Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:38:43.539272Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:38:43.539272Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:43.539272Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:43.539272Z", "running": 0, "size": 1}}] 2022-03-06T00:38:53.804 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:38:54.546 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:38:54.546 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:38:54.364777Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:38:54.364777Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:38:54.364777Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:38:54.364777Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:54.364777Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:54.364777Z", "running": 0, "size": 1}}] 2022-03-06T00:39:04.583 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:39:05.324 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:39:05.325 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:39:05.142886Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:39:05.142886Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:39:05.142886Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:39:05.142886Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:05.142886Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:05.142886Z", "running": 0, "size": 1}}] 2022-03-06T00:39:15.357 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:39:16.358 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:39:16.358 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:39:15.891964Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:39:15.891964Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:39:15.891964Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:39:15.891964Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:15.891964Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:15.891964Z", "running": 0, "size": 1}}] 2022-03-06T00:39:26.385 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:39:27.164 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:39:27.166 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:39:26.925370Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:39:26.925370Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:39:26.925370Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:39:26.925370Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:26.925370Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:26.925370Z", "running": 0, "size": 1}}] 2022-03-06T00:39:37.192 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:39:37.943 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:39:37.944 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:39:37.768865Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:39:37.768865Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:39:37.768865Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:39:37.768865Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:37.768865Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:37.768865Z", "running": 0, "size": 1}}] 2022-03-06T00:39:47.973 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:39:48.686 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:39:48.687 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:39:48.509525Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:39:48.509525Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:39:48.509525Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:39:48.509525Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:48.509525Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:48.509525Z", "running": 0, "size": 1}}] 2022-03-06T00:39:58.718 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:39:59.424 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:39:59.425 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:39:59.240035Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:39:59.240035Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:39:59.240035Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:39:59.240035Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:59.240035Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:59.240035Z", "running": 0, "size": 1}}] 2022-03-06T00:40:09.455 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:40:10.234 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:40:10.234 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:40:10.047938Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:40:10.047938Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:40:10.047938Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:40:10.047938Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:10.047938Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:10.047938Z", "running": 0, "size": 1}}] 2022-03-06T00:40:20.262 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:40:20.958 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:40:20.958 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:40:20.786012Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:40:20.786012Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:40:20.786012Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:40:20.786012Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:20.786012Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:20.786012Z", "running": 0, "size": 1}}] 2022-03-06T00:40:30.985 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:40:31.723 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:40:31.724 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:40:31.541929Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:40:31.541929Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:40:31.541929Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:40:31.541929Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:31.541929Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:31.541929Z", "running": 0, "size": 1}}] 2022-03-06T00:40:41.749 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:40:42.488 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:40:42.489 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:40:42.292210Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:40:42.292210Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:40:42.292210Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:40:42.292210Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:42.292210Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:42.292210Z", "running": 0, "size": 1}}] 2022-03-06T00:40:52.517 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:40:53.232 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:40:53.232 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:40:53.057659Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:40:53.057659Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:40:53.057659Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:40:53.057659Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:53.057659Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:53.057659Z", "running": 0, "size": 1}}] 2022-03-06T00:41:03.260 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:41:04.123 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:41:04.124 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:41:03.949038Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:41:03.949038Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:41:03.949038Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:41:03.949038Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:03.949038Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:03.949038Z", "running": 0, "size": 1}}] 2022-03-06T00:41:14.151 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:41:16.309 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:41:16.309 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:41:16.125794Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:41:16.125794Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:41:16.125794Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:41:16.125794Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:16.125794Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:16.125794Z", "running": 0, "size": 1}}] 2022-03-06T00:41:26.340 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:41:28.158 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:41:28.158 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:41:27.978447Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:41:27.978447Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:41:27.978447Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:41:27.978447Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:27.978447Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:27.978447Z", "running": 0, "size": 1}}] 2022-03-06T00:41:38.185 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:41:38.911 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:41:38.911 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:41:38.729619Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:41:38.729619Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:41:38.729619Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:41:38.729619Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:38.729619Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:38.729619Z", "running": 0, "size": 1}}] 2022-03-06T00:41:48.932 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:41:49.803 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:41:49.804 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:41:49.574022Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:41:49.574022Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:41:49.574022Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:41:49.574022Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:49.574022Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:49.574022Z", "running": 0, "size": 1}}] 2022-03-06T00:41:59.847 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:42:00.707 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:42:00.707 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:42:00.420979Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:42:00.420979Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:42:00.420979Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:42:00.420979Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:00.420979Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:00.420979Z", "running": 0, "size": 1}}] 2022-03-06T00:42:10.739 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:42:11.474 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:42:11.474 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:42:11.274994Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:42:11.274994Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:42:11.274994Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:42:11.274994Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:11.274994Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:11.274994Z", "running": 0, "size": 1}}] 2022-03-06T00:42:21.504 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:42:22.246 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:42:22.246 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:42:22.069589Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:42:22.069589Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:42:22.069589Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:42:22.069589Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:22.069589Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:22.069589Z", "running": 0, "size": 1}}] 2022-03-06T00:42:32.280 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:42:32.986 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:42:32.987 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:42:32.818536Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:42:32.818536Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:42:32.818536Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:42:32.818536Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:32.818536Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:32.818536Z", "running": 0, "size": 1}}] 2022-03-06T00:42:36.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:42:36.220974 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:42:36.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:42:36.870103 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:42:36.859+0000 7f00b3311380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:42:36.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:42:36.859+0000 7f00b3311380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:42:36.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:42:36.859+0000 7f00b3311380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:42:36.883 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:42:43.014 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:42:43.746 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:42:43.746 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:42:43.559218Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:42:43.559218Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:42:43.559218Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:42:43.559218Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:43.559218Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:43.559218Z", "running": 0, "size": 1}}] 2022-03-06T00:42:53.774 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:42:54.504 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:42:54.505 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:42:54.320494Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:42:54.320494Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:42:54.320494Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:42:54.320494Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:54.320494Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:54.320494Z", "running": 0, "size": 1}}] 2022-03-06T00:43:04.531 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:43:05.275 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:43:05.275 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:43:05.088586Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:43:05.088586Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:43:05.088586Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:43:05.088586Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:05.088586Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:05.088586Z", "running": 0, "size": 1}}] 2022-03-06T00:43:15.305 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:43:16.016 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:43:16.017 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:43:15.841347Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:43:15.841347Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:43:15.841347Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:43:15.841347Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:15.841347Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:15.841347Z", "running": 0, "size": 1}}] 2022-03-06T00:43:26.041 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:43:26.819 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:43:26.819 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:43:26.589472Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:43:26.589472Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:43:26.589472Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:43:26.589472Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:26.589472Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:26.589472Z", "running": 0, "size": 1}}] 2022-03-06T00:43:36.846 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:43:37.573 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:43:37.573 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:43:37.400309Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:43:37.400309Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:43:37.400309Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:43:37.400309Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:37.400309Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:37.400309Z", "running": 0, "size": 1}}] 2022-03-06T00:43:47.600 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:43:48.710 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:43:48.711 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:43:48.540445Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:43:48.540445Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:43:48.540445Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:43:48.540445Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:48.540445Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:48.540445Z", "running": 0, "size": 1}}] 2022-03-06T00:43:58.740 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:43:59.494 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:43:59.495 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:43:59.309444Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:43:59.309444Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:43:59.309444Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:43:59.309444Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:59.309444Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:59.309444Z", "running": 0, "size": 1}}] 2022-03-06T00:44:09.522 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:44:10.243 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:44:10.244 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:44:10.063959Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:44:10.063959Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:44:10.063959Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:44:10.063959Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:10.063959Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:10.063959Z", "running": 0, "size": 1}}] 2022-03-06T00:44:20.297 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:44:20.997 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:44:20.997 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:44:20.782109Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:44:20.782109Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:44:20.782109Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:44:20.782109Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:20.782109Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:20.782109Z", "running": 0, "size": 1}}] 2022-03-06T00:44:31.024 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:44:31.732 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:44:31.732 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:44:31.544256Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:44:31.544256Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:44:31.544256Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:44:31.544256Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:31.544256Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:31.544256Z", "running": 0, "size": 1}}] 2022-03-06T00:44:41.760 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:44:42.444 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:44:42.445 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:44:42.271847Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:44:42.271847Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:44:42.271847Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:44:42.271847Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:42.271847Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:42.271847Z", "running": 0, "size": 1}}] 2022-03-06T00:44:52.472 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:44:53.199 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:44:53.200 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:44:53.000717Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:44:53.000717Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:44:53.000717Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:44:53.000717Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:53.000717Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:53.000717Z", "running": 0, "size": 1}}] 2022-03-06T00:45:03.225 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:45:04.335 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:45:04.336 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:45:03.774229Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:45:03.774229Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:45:03.774229Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:45:03.774229Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:03.774229Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:03.774229Z", "running": 0, "size": 1}}] 2022-03-06T00:45:14.365 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:45:15.158 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:45:15.158 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:45:14.957793Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:45:14.957793Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:45:14.957793Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:45:14.957793Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:14.957793Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:14.957793Z", "running": 0, "size": 1}}] 2022-03-06T00:45:25.177 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:45:25.893 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:45:25.893 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:45:25.715399Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:45:25.715399Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:45:25.715399Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:45:25.715399Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:25.715399Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:25.715399Z", "running": 0, "size": 1}}] 2022-03-06T00:45:35.920 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:45:36.616 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:45:36.617 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:45:36.437500Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:45:36.437500Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:45:36.437500Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:45:36.437500Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:36.437500Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:36.437500Z", "running": 0, "size": 1}}] 2022-03-06T00:45:46.644 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:45:47.399 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:45:47.399 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:45:47.196916Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:45:47.196916Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:45:47.196916Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:45:47.196916Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:47.196916Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:47.196916Z", "running": 0, "size": 1}}] 2022-03-06T00:45:57.427 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:45:58.377 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:45:58.377 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:45:58.151733Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:45:58.151733Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:45:58.151733Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:45:58.151733Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:58.151733Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:58.151733Z", "running": 0, "size": 1}}] 2022-03-06T00:46:08.417 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:46:09.198 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:46:09.199 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:46:09.030501Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:46:09.030501Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:46:09.030501Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:46:09.030501Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:46:09.030501Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:46:09.030501Z", "running": 0, "size": 1}}] 2022-03-06T00:46:19.221 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:46:19.919 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:46:19.920 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:46:19.755321Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:46:19.755321Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:46:19.755321Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:46:19.755321Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:46:19.755321Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:46:19.755321Z", "running": 0, "size": 1}}] 2022-03-06T00:46:29.952 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:46:30.629 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:46:30.630 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:46:30.462715Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:46:30.462715Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:46:30.462715Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:46:30.462715Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:46:30.462715Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:46:30.462715Z", "running": 0, "size": 1}}] 2022-03-06T00:46:40.658 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:46:41.395 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:46:41.395 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:46:41.209465Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:46:41.209465Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:46:41.209465Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:46:41.209465Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:46:41.209465Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:46:41.209465Z", "running": 0, "size": 1}}] 2022-03-06T00:46:51.413 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:46:52.123 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:46:52.124 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:46:51.947366Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:46:51.947366Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:46:51.947366Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:46:51.947366Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:46:51.947366Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:46:51.947366Z", "running": 0, "size": 1}}] 2022-03-06T00:47:02.151 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:47:02.878 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:47:02.879 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:47:02.681531Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:47:02.681531Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:47:02.681531Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:47:02.681531Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:47:02.681531Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:47:02.681531Z", "running": 0, "size": 1}}] 2022-03-06T00:47:12.905 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:47:13.623 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:47:13.623 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:47:13.442349Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:47:13.442349Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:47:13.442349Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:47:13.442349Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:47:13.442349Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:47:13.442349Z", "running": 0, "size": 1}}] 2022-03-06T00:47:23.651 DEBUG:teuthology.orchestra.run.smithi018:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-hqzf2 -- ceph orch ls -f json 2022-03-06T00:47:24.434 INFO:teuthology.orchestra.run.smithi018.stdout: 2022-03-06T00:47:24.434 INFO:teuthology.orchestra.run.smithi018.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:37.000000Z", "last_refresh": "2022-03-06T00:47:24.179783Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:56.000000Z", "last_refresh": "2022-03-06T00:47:24.179783Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:06.000000Z", "last_refresh": "2022-03-06T00:47:24.179783Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:27:29.000000Z", "last_refresh": "2022-03-06T00:47:24.179783Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:47:24.179783Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:47:24.179783Z", "running": 0, "size": 1}}] 2022-03-06T00:47:24.462 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:47:24.463 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:47:24.464 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-06T00:47:24.464 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-06T00:47:24.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-06T00:47:24.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-06T00:47:24.511 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:47:24.512 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:47:24.537 DEBUG:teuthology.orchestra.remote:smithi018:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:47:24.552 DEBUG:teuthology.orchestra.run.smithi018:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-06T00:47:24.618 INFO:teuthology.orchestra.run.smithi018.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-06T00:47:24.627 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:47:24.628 DEBUG:teuthology.orchestra.run.smithi018:> kubectl delete -f cluster.yaml 2022-03-06T00:47:24.705 INFO:teuthology.orchestra.run.smithi018.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-06T00:47:24.709 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:24.692247 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-06T00:47:24.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:24.692479 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-06T00:47:24.712 INFO:tasks.rook.operator.smithi018.stdout:I0306 00:47:24.692757 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-06T00:47:24.713 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:24.692915 I | operator: successfully started the controller-runtime manager 2022-03-06T00:47:24.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:24.703256 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:47:24.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:24.703277 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:47:24.721 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:24.703298 I | operator: setting up schemes 2022-03-06T00:47:24.723 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:24.706075 I | operator: setting up the controller-runtime manager 2022-03-06T00:47:25.028 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.011248 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:47:25.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014384 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:47:25.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014434 I | ceph-cluster-controller: successfully started 2022-03-06T00:47:25.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014516 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:47:25.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014557 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:47:25.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014601 I | ceph-block-pool-controller: successfully started 2022-03-06T00:47:25.035 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014642 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:47:25.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014687 I | ceph-object-realm-controller: successfully started 2022-03-06T00:47:25.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014718 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:47:25.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014746 I | ceph-object-zone-controller: successfully started 2022-03-06T00:47:25.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.014935 I | ceph-object-controller: successfully started 2022-03-06T00:47:25.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015010 I | ceph-file-controller: successfully started 2022-03-06T00:47:25.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015070 I | ceph-nfs-controller: successfully started 2022-03-06T00:47:25.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015126 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:47:25.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015175 I | ceph-client-controller: successfully started 2022-03-06T00:47:25.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015213 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:47:25.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015269 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:47:25.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015306 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:47:25.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015343 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:47:25.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015382 I | ceph-bucket-topic: successfully started 2022-03-06T00:47:25.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015408 I | ceph-bucket-notification: successfully started 2022-03-06T00:47:25.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015437 I | ceph-bucket-notification: successfully started 2022-03-06T00:47:25.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.015464 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:47:25.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.016805 I | operator: starting the controller-runtime manager 2022-03-06T00:47:25.144 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.126763 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:47:25.144 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.126806 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:47:25.144 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.126814 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:47:25.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.130070 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:47:25.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.131708 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:25.229 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.211790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:25.239 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.222623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:25.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.817345 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:25.807+0000 7f7fe82c8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:25.835 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:25.807+0000 7f7fe82c8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:25.835 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:25.807+0000 7f7fe82c8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:25.835 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:25.866 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:25.849717 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:26.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:26.505904 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:26.495+0000 7fe4835b3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:26.524 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:26.495+0000 7fe4835b3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:26.524 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:26.495+0000 7fe4835b3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:26.524 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:26.558 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:26.541317 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:27.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:27.226019 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:27.215+0000 7ff0f621d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:27.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:27.215+0000 7ff0f621d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:27.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:27.215+0000 7ff0f621d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:27.244 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:27.270 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:27.253330 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:27.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:27.911418 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:27.903+0000 7f483af96380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:27.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:27.903+0000 7f483af96380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:27.930 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:27.903+0000 7f483af96380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:27.930 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:28.027 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:28.010614 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:28.677 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:28.659128 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:28.643+0000 7f9ba0fc9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:28.677 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:28.643+0000 7f9ba0fc9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:28.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:28.643+0000 7f9ba0fc9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:28.678 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:28.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:28.744395 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:29.516 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:29.497671 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:29.487+0000 7ffbfec6c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:29.517 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:29.487+0000 7ffbfec6c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:29.518 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:29.487+0000 7ffbfec6c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:29.518 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:29.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:29.664394 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:30.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:30.420469 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:30.411+0000 7fdca2699380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:30.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:30.411+0000 7fdca2699380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:30.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:30.411+0000 7fdca2699380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:30.438 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:30.766 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:30.748744 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:31.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:31.413597 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:31.403+0000 7ff8517b5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:31.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:31.403+0000 7ff8517b5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:31.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:31.403+0000 7ff8517b5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:31.431 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:32.080 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:32.062845 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:32.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:32.806849 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:32.799+0000 7fa2f4c99380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:32.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:32.799+0000 7fa2f4c99380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:32.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:32.799+0000 7fa2f4c99380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:32.825 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:34.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:34.093950 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:34.789 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:34.771683 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:34.763+0000 7fed485ad380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:34.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:34.763+0000 7fed485ad380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:34.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:34.763+0000 7fed485ad380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:34.791 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:35.433 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:35.416763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:35.446 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:35.429405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:37.356 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:37.339311 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:38.017 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:38.000652 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:37.991+0000 7faa79321380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:38.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:37.991+0000 7faa79321380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:38.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:37.991+0000 7faa79321380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:38.018 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:43.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:43.129521 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:43.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:43.786313 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:43.775+0000 7fda05e7b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:43.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:43.775+0000 7fda05e7b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:43.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:43.775+0000 7fda05e7b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:43.804 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:45.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:45.548341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:45.578 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:45.561349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:54.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:54.033153 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:47:54.885 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:54.868415 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:54.859+0000 7f25c446c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:54.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:54.859+0000 7f25c446c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:54.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:47:54.859+0000 7f25c446c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:54.886 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:55.647 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:55.630931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:55.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:47:55.642556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:05.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:05.750219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:05.780 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:05.763906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:15.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:15.386448 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:48:15.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:15.873503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:15.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:15.886123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:16.143 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:16.127299 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:48:16.115+0000 7fd557e4e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:48:16.668 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:48:16.115+0000 7fd557e4e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:48:16.668 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:48:16.115+0000 7fd557e4e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:48:16.669 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:48:26.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:26.016717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:26.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:26.029455 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:36.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:36.140951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:36.170 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:36.154126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:46.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:46.231347 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:46.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:46.242887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:56.455 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:56.438861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:56.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:56.451758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:57.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:57.098579 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:48:57.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:48:57.766676 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:48:57.759+0000 7fa6d230c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:48:57.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:48:57.759+0000 7fa6d230c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:48:57.784 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:48:57.759+0000 7fa6d230c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:48:57.784 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:49:06.584 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:06.568135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:06.597 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:06.581586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:16.671 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:16.655277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:16.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:16.670552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:26.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:26.793997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:26.827 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:26.807246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:36.916 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:36.900932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:36.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:36.913469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:46.984 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:46.968562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:46.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:46.979833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:57.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:57.088141 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:57.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:49:57.102902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:07.234 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:07.218584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:07.246 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:07.231604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:17.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:17.289896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:17.357 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:17.341653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:19.714 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:19.693048 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:50:20.488 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:20.473029 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:50:20.462+0000 7f79382dd380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:50:20.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:50:20.462+0000 7f79382dd380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:50:20.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:50:20.462+0000 7f79382dd380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:50:20.489 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:50:27.424 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:27.408655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:27.437 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:27.421848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:37.507 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:37.491848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:37.518 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:37.503039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:47.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:47.611697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:47.641 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:47.625945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:57.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:57.740272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:57.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:50:57.752740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:07.836 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:07.821154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:07.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:07.833552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:17.919 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:17.903793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:17.932 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:17.917675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:28.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:28.024385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:28.053 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:28.037921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:38.178 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:38.162472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:38.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:38.175864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:48.319 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:48.304259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:48.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:48.316868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:58.398 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:58.380344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:58.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:51:58.394051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:08.514 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:08.498759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:08.528 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:08.512272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:18.623 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:18.607651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:18.643 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:18.627138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:28.738 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:28.722944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:28.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:28.735454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:38.865 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:38.849741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:38.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:38.867362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:49.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:48.991730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:49.020 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:49.004124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:59.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:59.150140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:59.178 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:52:59.162932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:04.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:04.320291 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:53:05.090 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:05.073951 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:53:05.064+0000 7f9834bf2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:53:05.090 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:53:05.064+0000 7f9834bf2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:53:05.090 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:53:05.064+0000 7f9834bf2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:53:05.091 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:53:09.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:09.229000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:09.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:09.272540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:19.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:19.349671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:19.378 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:19.363373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:29.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:29.466129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:29.492 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:29.477502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:39.575 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:39.560208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:39.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:39.573822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:49.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:49.645892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:49.713 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:49.698704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:59.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:59.761631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:59.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:53:59.776486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:09.907 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:09.893500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:09.920 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:09.904413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:20.024 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:20.010762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:20.035 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:20.022131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:30.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:30.135415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:30.164 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:30.150865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:40.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:40.220390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:40.246 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:40.234030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:50.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:50.333847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:50.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:54:50.345859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:00.508 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:00.495931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:00.520 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:00.507850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:10.597 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:10.581265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:10.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:10.593933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:20.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:20.717693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:20.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:20.727976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:30.850 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:30.838549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:30.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:30.849534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:40.969 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:40.956892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:40.983 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:40.970390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:51.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:51.050638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:51.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:55:51.061983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:01.142 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:01.130287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:01.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:01.143009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:11.251 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:11.236449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:11.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:11.250867 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:21.370 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:21.358143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:21.382 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:21.369701 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:31.519 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:31.507483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:31.533 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:31.522075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:41.645 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:41.632984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:41.654 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:41.642697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:51.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:51.731828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:51.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:56:51.744091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:01.925 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:01.913809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:01.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:01.945686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:12.024 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:12.011643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:12.035 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:12.024124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:22.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:22.082498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:22.107 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:22.094717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:32.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:32.148143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:32.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:32.160031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:42.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:42.257381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:42.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:42.270648 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:52.404 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:52.393466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:52.418 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:57:52.407235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:02.521 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:02.510409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:02.536 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:02.525304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:12.624 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:12.613366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:12.647 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:12.635082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:22.713 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:22.702059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:22.726 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:22.715512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:32.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:32.762035 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T00:58:32.828 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:32.815253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:32.842 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:32.831214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:33.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:33.448581 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:58:33.437+0000 7f80aafb8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:58:33.463 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:58:33.437+0000 7f80aafb8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:58:33.463 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T00:58:33.437+0000 7f80aafb8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:58:33.463 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:58:42.911 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:42.899547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:42.972 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:42.960848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:53.063 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:53.052676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:53.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:58:53.062814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:03.142 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:03.131591 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:03.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:03.147923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:13.230 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:13.218234 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:13.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:13.232253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:23.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:23.299745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:23.322 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:23.310602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:33.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:33.456312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:33.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:33.467808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:43.601 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:43.581509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:43.604 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:43.593903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:53.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:53.674353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:53.699 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 00:59:53.687749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:03.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:03.742057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:03.765 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:03.754506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:13.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:13.818924 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:13.839 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:13.828820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:23.945 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:23.934516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:23.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:23.955046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:34.060 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:34.050016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:34.071 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:34.061536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:44.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:44.122999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:44.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:44.159452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:54.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:54.261733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:54.316 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:00:54.305749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:04.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:04.384460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:04.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:04.398510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:14.537 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:14.527772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:14.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:14.540723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:24.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:24.674676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:24.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:24.687880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:34.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:34.750478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:34.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:34.762845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:44.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:44.865942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:44.937 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:44.928264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:55.072 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:55.063353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:55.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:01:55.075655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:05.183 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:05.174555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:05.196 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:05.187438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:15.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:15.291550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:15.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:15.302468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:25.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:25.406991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:25.429 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:25.421088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:35.504 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:35.495512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:35.517 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:35.508448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:45.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:45.651232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:45.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:45.665249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:55.779 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:55.770873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:55.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:02:55.783082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:05.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:05.881002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:05.904 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:05.895546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:16.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:15.997770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:16.021 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:16.011299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:26.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:26.114332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:26.135 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:26.127248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:36.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:36.229488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:36.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:36.242083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:46.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:46.309873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:46.330 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:46.322691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:56.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:56.394033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:56.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:03:56.406546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:06.533 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:06.524874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:06.547 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:06.538702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:16.639 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:16.631431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:16.734 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:16.725836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:26.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:26.799442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:26.860 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:26.852313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:36.972 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:36.963835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:36.984 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:36.976796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:47.087 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:47.078734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:47.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:47.091763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:57.197 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:57.189225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:57.208 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:04:57.200851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:07.297 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:07.288866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:07.309 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:07.301440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:17.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:17.365313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:17.387 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:17.379282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:27.488 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:27.480914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:27.500 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:27.492620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:37.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:37.605378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:37.634 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:37.626205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:47.734 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:47.726976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:47.747 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:47.739973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:57.875 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:57.867936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:57.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:05:57.880820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:07.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:07.946912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:08.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:08.000600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:18.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:18.087464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:18.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:18.099126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:28.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:28.207619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:28.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:28.221813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:38.320 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:38.312481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:38.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:38.327557 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:48.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:48.399464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:48.418 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:48.411506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:58.547 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:58.540573 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:58.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:06:58.552750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:08.654 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:08.647553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:08.668 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:08.660769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:18.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:18.795970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:18.814 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:18.808459 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:29.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:28.995227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:29.017 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:29.011115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:39.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:39.078014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:39.098 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:39.091796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:49.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:49.179667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:49.198 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:49.191652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:59.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:59.281742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:59.315 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:07:59.308914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:09.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:09.407478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:09.426 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:09.420491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:19.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:19.519416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:19.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:19.534670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:29.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:29.796941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:29.812 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:29.806070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:39.867 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:39.861710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:39.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:39.882199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:50.101 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:50.095315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:50.114 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:08:50.108380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:00.206 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:00.199906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:00.218 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:00.212034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:10.281 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:10.275116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:10.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:10.320012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:20.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:20.396480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:20.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:20.408902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:28.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:28.818820 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T01:09:29.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:29.553805 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T01:09:29.542+0000 7fc96ce0e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T01:09:29.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T01:09:29.542+0000 7fc96ce0e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T01:09:29.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T01:09:29.542+0000 7fc96ce0e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T01:09:29.561 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T01:09:30.491 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:30.485537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:30.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:30.495529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:40.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:40.557113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:40.574 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:40.568756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:50.641 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:50.635995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:50.711 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:09:50.705276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:00.808 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:00.802542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:00.821 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:00.815969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:10.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:10.888502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:10.907 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:10.901487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:21.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:21.007244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:21.025 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:21.019947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:31.086 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:31.080797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:31.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:31.122468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:41.193 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:41.187777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:41.204 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:41.198907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:51.376 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:51.370945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:51.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:10:51.383399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:01.455 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:01.449958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:01.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:01.461941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:11.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:11.547432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:11.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:11.623669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:21.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:21.739223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:21.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:21.750613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:31.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:31.861778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:31.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:31.873952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:42.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:42.012373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:42.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:42.026799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:52.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:52.094250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:52.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:11:52.106350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:02.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:02.183032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:02.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:02.199352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:12.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:12.270825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:12.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:12.282779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:22.361 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:22.355369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:22.374 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:22.368508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:32.454 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:32.448801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:32.509 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:32.503839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:42.655 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:42.649425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:42.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:42.662401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:52.718 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:52.711784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:52.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:12:52.722285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:02.855 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:02.850146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:02.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:02.864528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:12.981 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:12.975492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:12.992 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:12.987153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:23.112 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:23.106708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:23.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:23.117148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:33.230 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:33.224650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:33.245 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:33.238692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:43.316 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:43.309684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:43.329 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:43.323798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:53.439 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:53.434118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:53.448 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:13:53.443225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:03.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:03.555590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:03.575 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:03.569971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:13.657 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:13.651407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:13.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:13.663888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:23.774 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:23.768369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:23.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:23.777609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:33.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:33.896457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:33.912 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:33.906802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:44.064 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:44.057999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:44.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:44.070921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:54.134 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:54.127818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:54.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:14:54.139875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:04.207 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:04.201336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:04.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:04.250092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:14.323 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:14.317319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:14.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:14.330119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:24.398 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:24.392125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:24.411 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:24.405019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:34.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:34.471820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:34.490 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:34.484224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:44.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:44.558698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:44.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:44.572798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:54.683 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:54.670711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:54.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:15:54.680262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:04.784 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:04.777615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:04.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:04.790839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:14.865 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:14.859538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:14.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:14.873732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:25.005 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:24.999478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:25.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:25.012639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:35.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:35.139597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:35.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:35.153436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:45.219 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:45.212078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:45.230 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:45.224640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:55.382 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:55.376511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:55.393 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:16:55.387898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:05.464 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:05.457968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:05.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:05.471749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:15.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:15.562998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:15.581 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:15.575742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:25.656 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:25.649486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:25.714 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:25.708985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:35.784 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:35.778891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:35.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:35.790823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:45.863 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:45.857783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:45.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:45.870291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:55.985 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:55.980476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:56.001 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:17:55.996238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:06.113 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:06.108538 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:06.125 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:06.120499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:16.182 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:16.174615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:16.195 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:16.189725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:26.267 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:26.261619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:26.735 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:26.275575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:36.352 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:36.346743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:36.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:36.359377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:46.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:46.425163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:46.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:46.476768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:56.626 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:56.621675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:56.639 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:18:56.634297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:06.727 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:06.720583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:06.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:06.730312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:16.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:16.824461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:16.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:16.838752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:26.926 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:26.921859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:26.938 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:26.933657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:37.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:37.034749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:37.056 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:37.050096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:47.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:47.124883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:47.143 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:47.138140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:57.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:57.266670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:57.283 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:19:57.278122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:07.370 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:07.365904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:07.381 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:07.376820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:17.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:17.476956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:17.498 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:17.492751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:27.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:27.584024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:27.601 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:27.597201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:37.712 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:37.707614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:37.727 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:37.722799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:47.826 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:47.821953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:47.840 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:47.835515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:57.970 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:57.965718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:57.983 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:20:57.979251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:08.088 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:08.083495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:08.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:08.098708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:18.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:18.243279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:18.261 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:18.256856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:28.351 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:28.347016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:28.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:28.359075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:38.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:38.455193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:38.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:38.466880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:48.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:48.715309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:48.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:48.728813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:58.937 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:58.932416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:58.970 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:21:58.965707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:09.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:09.078239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:09.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:09.089463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:19.169 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:19.165172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:19.180 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:19.176574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:29.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:29.267042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:29.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:29.279660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:39.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:39.380857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:39.397 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:39.392836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:49.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:49.495875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:49.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:49.508424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:59.575 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:59.570849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:59.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:22:59.583662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:09.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:09.681753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:09.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:09.693853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:19.827 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:19.822684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:19.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:19.834859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:29.999 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:29.995672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:30.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:30.009686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:40.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:40.100580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:40.117 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:40.113477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:50.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:50.222979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:50.240 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:23:50.236554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:00.357 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:00.353151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:00.369 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:00.365875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:10.471 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:10.467884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:10.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:10.481602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:20.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:20.539589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:20.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:20.556085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:30.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:30.663611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:30.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:30.677530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:40.794 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:40.790459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:40.806 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:40.802835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:50.905 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:50.896764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:50.912 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:24:50.909132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:01.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:01.081415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:01.098 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:01.094976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:11.176 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:11.173564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:11.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:11.184688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:21.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:21.297693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:21.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:21.309644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:31.434 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:31.431304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:31.445 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:31.443128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:41.509 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:41.506490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:41.522 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:41.519313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:51.665 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:51.662709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:51.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:25:51.675103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:01.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:01.755187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:01.771 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:01.767343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:09.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:09.562877 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T01:26:10.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:10.297735 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T01:26:10.287+0000 7fcc370ba380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T01:26:10.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T01:26:10.287+0000 7fcc370ba380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T01:26:10.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T01:26:10.287+0000 7fcc370ba380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T01:26:10.302 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T01:26:11.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:11.876361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:11.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:11.889325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:21.999 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:21.996342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:22.009 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:22.006310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:32.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:32.065210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:32.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:32.142997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:42.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:42.241205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:42.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:42.256776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:52.322 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:52.319132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:52.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:26:52.330172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:02.418 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:02.415749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:02.435 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:02.428770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:12.525 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:12.522781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:12.538 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:12.535792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:22.600 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:22.598216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:22.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:22.611587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:32.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:32.726233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:32.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:32.738359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:42.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:42.860380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:42.873 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:42.871515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:52.969 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:52.967450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:52.983 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:27:52.981360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:03.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:03.046185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:03.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:03.058566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:13.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:13.127747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:13.140 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:13.139013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:23.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:23.246248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:23.263 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:23.259498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:33.371 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:33.366758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:33.380 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:33.377463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:43.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:43.543993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:43.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:43.557344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:53.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:53.717748 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:53.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:28:53.731411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:03.865 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:03.863782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:03.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:03.876662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:14.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:14.002325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:14.221 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:14.014238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:24.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:24.114079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:24.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:24.126585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:34.197 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:34.196055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:34.208 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:34.207720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:44.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:44.324446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:44.337 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:44.336821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:54.405 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:54.404805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:54.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:29:54.464059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:04.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:04.543809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:04.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:04.557879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:14.628 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:14.626060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:14.638 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:14.638033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:24.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:24.744632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:24.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:24.758180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:34.818 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:34.817592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:34.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:34.830177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:44.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:44.957617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:44.971 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:44.971274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:55.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:55.047750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:55.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:30:55.061764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:05.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:05.188505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:05.201 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:05.201020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:15.334 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:15.333947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:15.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:15.348574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:25.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:25.484983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:25.500 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:25.499900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:35.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:35.633183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:35.646 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:35.646604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:45.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:45.751295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:45.767 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:45.766282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:55.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:55.868626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:55.914 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:31:55.913726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:05.993 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:05.992943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:06.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:06.005933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:16.118 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:16.118712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:16.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:16.130412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:26.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:26.237925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:26.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:26.255653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:36.382 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:36.382893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:36.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:36.393511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:46.591 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:46.590671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:46.612 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:46.603859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:56.705 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:56.698607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:56.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:32:56.717694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:06.850 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:06.850633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:06.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:06.862609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:16.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:16.946997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:16.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:16.958180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:27.064 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:27.064660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:27.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:27.077784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:37.201 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:37.201092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:37.214 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:37.214893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:47.321 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:47.320932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:47.333 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:47.333243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:57.396 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:57.395641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:57.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:33:57.407728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:07.521 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:07.521709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:07.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:07.535485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:17.598 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:17.598656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:17.613 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:17.613976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:27.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:27.769387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:27.780 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:27.781507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:37.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:37.883672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:37.897 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:37.896981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:48.031 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:48.031579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:48.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:48.045496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:58.164 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:58.164084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:58.176 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:34:58.177059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:08.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:08.235982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:08.246 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:08.247076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:18.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:18.313577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:18.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:18.346847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:28.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:28.459668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:28.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:28.481525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:38.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:38.563663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:38.577 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:38.578053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:48.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:48.660574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:48.672 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:48.673090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:58.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:58.773910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:58.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:35:58.785799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:08.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:08.872198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:08.924 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:08.923985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:19.024 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:19.025055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:19.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:19.038048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:29.142 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:29.141891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:29.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:29.153908 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:39.214 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:39.214148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:39.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:39.226672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:49.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:49.331918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:49.343 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:49.344065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:59.443 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:59.443358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:59.456 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:36:59.457013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:09.556 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:09.556070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:09.570 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:09.570725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:19.645 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:19.645272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:19.656 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:19.656562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:29.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:29.805292 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:29.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:29.818243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:39.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:39.910569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:39.920 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:39.920038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:50.035 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:50.036096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:50.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:37:50.048450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:00.112 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:00.112314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:00.124 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:00.125040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:10.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:10.250273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:10.262 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:10.262993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:20.359 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:20.360031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:20.372 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:20.372495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:30.446 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:30.446766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:30.533 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:30.533642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:40.650 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:40.650785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:40.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:40.660866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:50.741 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:50.741033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:50.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:38:50.753983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:00.819 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:00.816585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:00.831 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:00.831805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:10.941 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:10.942014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:10.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:10.954871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:21.035 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:21.035296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:21.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:21.046325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:31.118 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:31.117680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:31.167 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:31.166214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:41.231 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:41.231229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:41.245 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:41.244957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:51.314 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:51.312995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:51.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:39:51.324864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:01.419 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:01.418548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:01.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:01.430080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:11.542 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:11.540955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:11.554 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:11.553249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:21.666 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:21.664634 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:21.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:21.677768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:31.759 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:31.757261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:31.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:31.801176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:41.930 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:41.925671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:41.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:41.944494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:52.045 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:52.042473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:52.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:40:52.055187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:02.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:02.172548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:02.187 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:02.184202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:12.299 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:12.296139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:12.356 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:12.308885 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:22.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:22.383254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:22.452 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:22.449520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:32.546 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:32.542724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:32.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:32.556401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:42.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:42.657571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:42.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:42.670708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:52.780 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:52.775772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:52.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:41:52.786802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:02.856 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:02.852075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:02.867 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:02.863119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:13.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:13.086348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:13.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:13.099181 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:23.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:23.384791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:23.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:23.397526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:33.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:33.548045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:33.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:33.560813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:43.666 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:43.660851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:43.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:43.673813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:50.327 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:50.321345 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T01:42:51.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:51.099370 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T01:42:51.087+0000 7fc6e569d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T01:42:51.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T01:42:51.087+0000 7fc6e569d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T01:42:51.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T01:42:51.087+0000 7fc6e569d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T01:42:51.106 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T01:42:53.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:53.790579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:53.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:42:53.803462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:03.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:03.876251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:03.945 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:03.938424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:14.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:14.043204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:14.063 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:14.057302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:24.163 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:24.157070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:24.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:24.169194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:34.260 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:34.253561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:34.273 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:34.266496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:44.342 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:44.335305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:44.354 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:44.347767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:54.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:54.423732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:54.440 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:43:54.433983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:04.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:04.535771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:04.556 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:04.549041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:14.649 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:14.641743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:14.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:14.653838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:24.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:24.750325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:24.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:24.762989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:34.875 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:34.867148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:34.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:34.880650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:44.956 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:44.949098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:45.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:45.004612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:55.071 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:55.063841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:55.085 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:44:55.078130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:05.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:05.147311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:05.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:05.159129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:15.275 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:15.267704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:15.287 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:15.279740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:25.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:25.384159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:25.416 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:25.409256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:35.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:35.544380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:35.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:35.559706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:45.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:45.675737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:45.700 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:45.689203 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:55.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:55.796478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:55.818 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:45:55.810264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:05.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:05.895728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:05.915 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:05.908313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:15.988 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:15.980781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:16.001 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:15.993569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:26.064 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:26.056067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:26.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:26.068756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:36.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:36.153746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:36.268 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:36.258941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:46.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:46.376791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:46.398 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:46.390777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:56.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:56.457657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:56.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:46:56.469510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:06.572 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:06.563078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:06.585 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:06.577094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:16.646 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:16.637741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:16.701 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:16.693157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:26.822 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:26.814233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:26.835 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:26.826281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:36.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:36.934828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:36.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:36.948000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:47.079 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:47.071129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:47.090 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:47.082614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:57.221 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:57.213575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:57.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:47:57.224089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:07.376 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:07.368428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:07.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:07.378307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:17.453 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:17.445010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:17.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:17.457845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:27.607 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:27.598825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:27.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:27.611930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:37.695 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:37.686716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:37.707 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:37.699037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:47.814 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:47.805233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:47.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:47.817641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:57.921 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:57.912342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:57.933 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:48:57.924576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:08.075 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:08.067083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:08.089 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:08.080823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:18.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:18.140935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:18.162 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:18.153825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:28.343 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:28.334930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:28.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:28.396629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:38.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:38.531081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:38.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:38.545038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:48.692 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:48.683720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:48.702 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:48.694155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:58.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:58.761978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:58.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:49:58.776735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:08.863 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:08.854781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:08.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:08.917961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:18.999 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:18.990239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:19.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:19.004305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:29.108 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:29.099802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:29.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:29.113162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:39.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:39.247503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:39.270 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:39.261124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:49.397 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:49.388707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:49.437 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:49.418880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:59.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:59.535805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:59.560 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:50:59.551472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:09.647 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:09.637623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:09.711 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:09.701896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:19.857 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:19.848009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:19.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:19.860650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:29.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:29.985922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:30.008 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:29.999112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:40.150 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:40.141412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:40.165 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:40.156090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:50.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:50.225398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:50.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:51:50.237640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:00.315 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:00.305466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:00.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:00.316476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:10.391 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:10.382528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:10.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:10.408917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:20.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:20.477799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:20.530 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:20.521306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:30.626 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:30.616571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:30.639 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:30.629909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:40.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:40.729807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:40.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:40.741537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:50.815 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:50.806034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:50.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:52:50.862471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:00.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:00.957878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:00.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:00.969569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:11.089 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:11.079363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:11.101 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:11.091991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:21.163 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:21.153340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:21.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:21.165083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:31.290 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:31.279857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:31.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:31.292322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:41.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:41.404555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:41.427 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:41.417931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:51.534 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:51.524300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:51.546 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:53:51.536877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:01.704 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:01.694167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:01.718 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:01.708467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:11.821 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:11.810833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:11.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:11.824898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:21.908 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:21.898519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:21.919 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:21.909483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:32.022 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:32.011646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:32.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:32.023505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:42.105 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:42.094824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:42.117 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:42.107542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:52.200 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:52.189972 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:52.221 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:54:52.210978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:02.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:02.325345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:02.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:02.337869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:12.472 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:12.461287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:12.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:12.474979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:22.541 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:22.530795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:22.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:22.542470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:32.625 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:32.614627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:32.640 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:32.629306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:42.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:42.732984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:42.758 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:42.747340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:52.892 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:52.881503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:52.905 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:55:52.894445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:02.978 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:02.966975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:02.992 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:02.981575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:13.061 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:13.050100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:13.125 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:13.114017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:23.225 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:23.214752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:23.241 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:23.230069 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:33.352 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:33.341338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:33.366 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:33.355825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:43.518 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:43.506958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:43.530 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:43.519755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:53.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:53.617940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:53.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:56:53.630882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:03.715 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:03.703276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:03.728 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:03.717046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:13.840 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:13.828705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:13.854 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:13.843105 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:23.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:23.946147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:23.972 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:23.960857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:34.038 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:34.027022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:34.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:34.039799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:44.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:44.118705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:44.144 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:44.132992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:54.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:54.245231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:54.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:57:54.259805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:04.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:04.395082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:04.419 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:04.408390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:14.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:14.515855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:14.540 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:14.528284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:24.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:24.598978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:24.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:24.657342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:34.741 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:34.729732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:34.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:34.741646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:44.852 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:44.840075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:44.864 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:44.853114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:54.966 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:54.954304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:55.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:58:55.006080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:05.070 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:05.057614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:05.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:05.070570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:15.184 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:15.172259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:15.198 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:15.186228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:25.314 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:25.301388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:25.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:25.313533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:31.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:31.107776 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T01:59:31.865 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:31.853187 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T01:59:31.840+0000 7f7869dc8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T01:59:31.866 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T01:59:31.840+0000 7f7869dc8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T01:59:31.866 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T01:59:31.840+0000 7f7869dc8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T01:59:31.866 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T01:59:35.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:35.378521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:35.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:35.391066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:45.503 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:45.490956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:45.514 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:45.502319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:55.574 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:55.561826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:55.586 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 01:59:55.574033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:05.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:05.641196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:05.666 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:05.653516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:15.741 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:15.728545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:15.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:15.741558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:25.859 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:25.846891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:25.873 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:25.860538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:35.945 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:35.932760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:35.959 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:35.946926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:46.066 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:46.053163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:46.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:46.063141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:56.206 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:56.193146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:56.219 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:00:56.206892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:06.281 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:06.268899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:06.294 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:06.281255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:16.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:16.351543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:16.379 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:16.365333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:26.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:26.474730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:26.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:26.486188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:36.609 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:36.596136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:36.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:36.608413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:46.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:46.716751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:46.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:46.727308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:56.899 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:56.885147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:56.913 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:01:56.894712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:06.974 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:06.961239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:06.986 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:06.973074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:17.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:17.062943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:17.089 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:17.075898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:27.160 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:27.146381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:27.173 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:27.160448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:37.269 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:37.255977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:37.281 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:37.268641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:47.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:47.334491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:47.367 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:47.354084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:57.494 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:57.481057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:57.509 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:02:57.494725 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:07.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:07.604678 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:07.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:07.615823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:17.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:17.771203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:17.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:17.785395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:27.913 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:27.899680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:27.925 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:27.912090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:38.000 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:37.986171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:38.014 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:38.000787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:48.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:48.117631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:48.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:48.130970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:58.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:58.203605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:58.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:03:58.262136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:08.372 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:08.358612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:08.383 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:08.369157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:18.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:18.484985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:18.511 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:18.496773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:28.572 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:28.558522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:28.584 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:28.569953 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:38.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:38.639239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:38.665 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:38.651337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:48.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:48.707842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:48.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:48.716402 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:58.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:58.816148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:58.934 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:04:58.920100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:09.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:09.038116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:09.064 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:09.049858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:19.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:19.118464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:19.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:19.131360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:29.229 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:29.214969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:29.241 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:29.226918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:39.341 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:39.326188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:39.354 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:39.340049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:49.426 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:49.411548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:49.438 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:49.423488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:59.542 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:59.527413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:59.555 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:05:59.540686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:09.709 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:09.694348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:09.721 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:09.706223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:19.881 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:19.866006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:19.896 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:19.881643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:30.023 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:30.008524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:30.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:30.018410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:40.141 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:40.126493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:40.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:40.138320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:50.252 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:50.237371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:50.266 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:06:50.251176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:00.378 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:00.363240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:00.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:00.378579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:10.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:10.496656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:10.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:10.508115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:20.594 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:20.579000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:20.652 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:20.637001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:30.731 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:30.715509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:30.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:30.728350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:40.839 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:40.823592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:40.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:40.887560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:51.001 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:50.985313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:51.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:07:50.996689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:01.138 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:01.122575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:01.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:01.155827 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:11.253 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:11.236553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:11.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:11.247834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:21.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:21.311659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:21.342 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:21.327022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:31.410 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:31.394511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:31.492 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:31.476986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:41.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:41.552945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:41.592 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:41.576351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:51.709 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:51.693225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:51.725 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:08:51.709626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:01.791 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:01.775862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:01.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:01.788228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:11.872 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:11.856153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:11.920 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:11.904298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:22.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:21.987715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:22.067 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:22.051974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:32.156 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:32.140636 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:32.167 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:32.152139 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:42.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:42.285410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:42.314 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:42.298667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:52.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:52.391047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:52.420 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:09:52.404999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:02.504 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:02.488221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:02.516 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:02.500993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:12.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:12.604631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:12.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:12.618251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:22.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:22.817708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:22.847 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:22.831446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:32.948 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:32.933086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:32.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:32.950113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:43.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:43.041860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:43.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:43.052601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:53.162 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:53.146441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:53.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:10:53.159815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:03.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:03.289876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:03.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:03.302489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:13.388 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:13.372602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:13.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:13.384782 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:23.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:23.523589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:23.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:23.537355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:33.701 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:33.685428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:33.715 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:33.699337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:43.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:43.768088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:43.846 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:43.831064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:53.968 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:53.952557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:53.981 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:11:53.965103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:04.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:04.067996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:04.101 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:04.085540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:14.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:14.175384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:14.221 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:14.205751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:24.341 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:24.325896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:24.354 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:24.339146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:34.420 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:34.404789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:34.436 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:34.417250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:44.529 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:44.513039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:44.542 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:44.526372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:54.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:54.617873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:54.647 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:12:54.630111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:04.712 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:04.696726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:04.729 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:04.713422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:14.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:14.808549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:14.835 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:14.819211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:24.948 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:24.932146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:24.964 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:24.948255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:35.072 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:35.057130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:35.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:35.069163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:45.181 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:45.162491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:45.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:45.174031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:55.293 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:55.278703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:55.308 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:13:55.293050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:05.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:05.385618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:05.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:05.400150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:15.520 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:15.506073 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:15.532 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:15.518166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:25.658 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:25.643085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:25.666 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:25.652552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:35.782 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:35.767768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:35.793 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:35.778985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:45.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:45.864756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:45.891 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:45.877248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:56.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:55.993456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:56.021 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:14:56.006998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:06.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:06.113675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:06.139 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:06.125484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:16.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:16.191628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:16.219 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:16.205026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:26.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:26.275352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:26.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:26.286940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:36.399 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:36.385624 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:36.411 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:36.397922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:46.475 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:46.461816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:46.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:46.473898 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:56.598 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:56.585745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:56.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:15:56.598749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:06.726 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:06.713249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:06.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:06.723700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:11.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:11.861112 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T02:16:12.566 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:12.552966 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T02:16:12.543+0000 7fade2243380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T02:16:12.566 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T02:16:12.543+0000 7fade2243380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T02:16:12.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T02:16:12.543+0000 7fade2243380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T02:16:12.567 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T02:16:16.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:16.830079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:16.854 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:16.841928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:26.924 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:26.911780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:26.936 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:26.924216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:37.011 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:36.999047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:37.024 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:37.011552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:47.087 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:47.074622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:47.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:47.087457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:57.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:57.178506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:57.204 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:16:57.192027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:07.320 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:07.307860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:07.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:07.318571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:17.447 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:17.434854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:17.459 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:17.447149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:27.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:27.523276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:27.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:27.537809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:37.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:37.647730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:37.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:37.662157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:47.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:47.849766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:47.873 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:47.861555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:58.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:58.069780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:58.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:17:58.082372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:08.283 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:08.271525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:08.296 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:08.284520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:18.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:18.396548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:18.421 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:18.410244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:28.515 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:28.503623 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:28.529 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:28.512996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:38.648 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:38.637386 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:38.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:38.650512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:48.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:48.763507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:48.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:48.774769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:58.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:58.826988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:58.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:18:58.876784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:08.958 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:08.947750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:08.972 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:08.961042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:19.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:19.088217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:19.113 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:19.102547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:29.222 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:29.211488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:29.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:29.226010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:39.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:39.291774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:39.314 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:39.304068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:49.372 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:49.361807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:49.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:49.375108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:59.491 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:59.480503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:59.504 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:19:59.493361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:09.649 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:09.639001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:09.662 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:09.651967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:19.748 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:19.737600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:19.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:19.749475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:29.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:29.819379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:29.841 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:29.831234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:39.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:39.899713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:39.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:39.913104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:50.025 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:50.015178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:50.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:20:50.026741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:00.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:00.109313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:00.132 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:00.122266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:10.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:10.195396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:10.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:10.207952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:20.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:20.326416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:20.354 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:20.344453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:30.418 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:30.408440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:30.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:30.420221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:40.535 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:40.522092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:40.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:40.533504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:50.611 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:50.601640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:50.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:21:50.687331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:00.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:00.760409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:00.782 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:00.772604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:10.862 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:10.853148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:10.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:10.865311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:20.990 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:20.980649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:21.033 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:21.011083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:31.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:31.101639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:31.126 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:31.116917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:41.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:41.225982 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:41.253 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:41.244278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:51.338 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:51.328857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:51.350 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:22:51.340994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:01.442 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:01.432828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:01.454 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:01.445497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:11.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:11.512986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:11.534 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:11.525247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:21.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:21.621153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:21.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:21.632751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:31.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:31.724627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:31.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:31.736980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:41.889 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:41.880582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:41.901 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:41.892435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:52.004 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:51.995729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:52.026 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:23:52.017256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:02.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:02.166086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:02.191 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:02.182104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:12.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:12.309075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:12.372 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:12.362188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:22.462 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:22.452947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:22.474 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:22.465233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:32.540 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:32.531049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:32.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:32.544481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:42.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:42.652535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:42.676 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:42.667697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:52.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:52.728840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:52.805 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:24:52.796443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:02.873 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:02.864952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:02.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:02.877166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:12.990 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:12.981874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:13.004 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:12.995729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:23.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:23.106337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:23.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:23.120579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:33.195 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:33.187260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:33.208 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:33.199476 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:43.273 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:43.261453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:43.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:43.273862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:53.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:53.338250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:53.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:25:53.350150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:03.475 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:03.466893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:03.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:03.478990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:13.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:13.579106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:13.601 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:13.592624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:23.717 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:23.708725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:23.729 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:23.720704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:33.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:33.821540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:33.845 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:33.837386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:43.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:43.951753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:43.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:43.967047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:54.040 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:54.032041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:54.056 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:26:54.045601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:04.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:04.114370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:04.135 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:04.126465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:14.203 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:14.195433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:14.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:14.209209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:24.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:24.303501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:24.324 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:24.316353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:34.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:34.391945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:34.414 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:34.406065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:44.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:44.473436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:44.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:44.485753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:54.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:54.554300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:54.575 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:27:54.567050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:04.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:04.677206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:04.696 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:04.688838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:14.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:14.820556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:14.866 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:14.857554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:24.955 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:24.947133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:24.968 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:24.959775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:35.053 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:35.045548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:35.065 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:35.057257 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:45.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:45.158123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:45.178 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:45.170706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:55.287 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:55.277691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:55.299 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:28:55.290749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:05.361 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:05.352938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:05.372 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:05.364400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:15.439 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:15.430948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:15.528 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:15.520078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:25.608 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:25.599957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:25.677 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:25.669537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:35.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:35.808801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:35.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:35.821475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:45.944 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:45.936532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:45.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:45.950268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:56.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:56.028252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:56.080 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:29:56.072492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:06.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:06.121214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:06.141 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:06.133194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:16.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:16.263414 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:16.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:16.276585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:26.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:26.550546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:26.571 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:26.563817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:36.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:36.743712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:36.830 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:36.822545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:46.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:46.946812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:46.969 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:46.961453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:57.289 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:57.281829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:57.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:30:57.295339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:07.369 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:07.361451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:07.381 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:07.373691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:17.548 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:17.539892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:17.561 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:17.553171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:27.626 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:27.618230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:27.637 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:27.630186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:37.704 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:37.696554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:37.715 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:37.707683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:47.784 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:47.776130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:47.836 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:47.828189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:57.911 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:57.903363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:57.986 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:31:57.916445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:08.043 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:08.035896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:08.059 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:08.050982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:18.193 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:18.185666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:18.206 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:18.198288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:28.367 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:28.359943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:28.378 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:28.370538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:38.453 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:38.445679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:38.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:38.457178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:48.625 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:48.617041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:48.637 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:48.629495 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:52.568 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:52.560446 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T02:32:53.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:53.235089 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T02:32:53.220+0000 7f1870710380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T02:32:53.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T02:32:53.220+0000 7f1870710380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T02:32:53.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T02:32:53.220+0000 7f1870710380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T02:32:53.244 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T02:32:58.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:58.702277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:58.722 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:32:58.715192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:08.788 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:08.780672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:08.806 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:08.795987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:18.906 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:18.898629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:18.918 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:18.910861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:28.976 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:28.968612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:28.989 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:28.981177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:39.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:39.084319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:39.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:39.097037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:49.203 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:49.195248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:49.217 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:49.210062 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:59.289 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:59.281212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:59.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:33:59.297338 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:09.417 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:09.409673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:09.429 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:09.421857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:19.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:19.489204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:19.508 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:19.500890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:29.604 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:29.596313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:29.617 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:29.609106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:39.718 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:39.709915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:39.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:39.722431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:49.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:49.840986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:49.864 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:49.856144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:59.932 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:59.924052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:00.001 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:34:59.993609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:10.114 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:10.106498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:10.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:10.119888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:20.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:20.228488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:20.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:20.240946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:30.353 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:30.345161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:30.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:30.356438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:40.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:40.493553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:40.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:40.504663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:50.610 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:50.602642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:50.625 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:35:50.615945 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:00.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:00.712016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:00.775 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:00.768034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:10.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:10.874719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:10.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:10.887749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:20.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:20.982375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:21.004 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:20.996884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:31.065 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:31.055492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:31.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:31.066915 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:41.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:41.138152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:41.159 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:41.151108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:51.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:51.212608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:51.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:36:51.227711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:01.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:01.296874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:01.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:01.310037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:11.390 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:11.381913 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:11.402 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:11.394888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:21.464 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:21.455852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:21.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:21.469059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:31.615 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:31.606915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:31.630 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:31.622093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:41.711 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:41.703087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:41.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:41.716549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:51.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:51.776927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:51.840 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:37:51.832019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:01.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:01.938973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:01.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:01.952145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:12.052 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:12.044571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:12.067 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:12.059343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:22.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:22.122510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:22.179 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:22.171272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:32.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:32.235755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:32.315 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:32.307133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:42.420 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:42.412248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:42.434 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:42.426463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:52.506 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:52.497940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:52.518 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:38:52.510399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:02.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:02.587361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:02.607 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:02.599536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:12.677 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:12.669098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:12.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:12.702606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:22.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:22.791842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:22.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:22.829690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:32.948 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:32.939849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:32.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:32.953219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:43.055 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:43.047801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:43.067 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:43.059117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:53.192 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:53.184208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:53.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:39:53.196620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:03.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:03.263496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:03.283 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:03.275623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:13.362 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:13.354659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:13.376 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:13.368682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:23.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:23.479065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:23.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:23.491904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:33.601 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:33.593741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:33.615 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:33.608409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:43.704 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:43.696676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:43.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:43.708681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:53.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:53.821804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:53.841 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:40:53.834392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:03.916 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:03.908629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:03.970 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:03.963500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:14.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:14.043065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:14.063 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:14.055681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:24.143 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:24.135892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:24.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:24.149055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:34.221 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:34.213824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:34.285 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:34.277923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:44.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:44.399191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:44.417 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:44.410058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:54.509 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:54.502398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:54.522 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:41:54.515524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:04.597 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:04.590065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:04.610 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:04.602796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:14.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:14.672968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:14.692 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:14.685960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:24.788 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:24.780833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:24.800 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:24.793955 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:34.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:34.867393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:35.442 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:34.916678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:44.982 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:44.975413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:44.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:44.988489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:55.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:55.095029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:55.115 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:42:55.108301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:05.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:05.209071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:05.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:05.222020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:15.292 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:15.285662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:15.304 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:15.297792 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:25.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:25.366787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:25.386 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:25.380211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:35.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:35.539004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:35.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:35.551048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:45.677 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:45.671203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:45.690 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:45.684584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:55.763 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:55.757277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:55.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:43:55.818202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:05.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:05.882245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:05.902 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:05.895972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:15.977 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:15.971290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:15.992 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:15.984748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:26.079 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:26.073615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:26.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:26.087832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:36.164 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:36.157683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:36.178 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:36.171970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:46.251 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:46.245229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:46.262 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:46.256416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:56.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:56.329535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:56.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:44:56.342437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:06.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:06.472445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:06.494 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:06.488540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:16.600 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:16.594394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:16.612 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:16.606517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:26.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:26.733728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:26.753 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:26.747078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:36.818 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:36.812700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:36.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:36.865106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:46.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:46.923474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:46.938 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:46.932910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:57.044 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:57.038252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:57.056 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:45:57.051276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:07.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:07.183935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:07.202 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:07.196585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:17.323 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:17.317383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:17.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:17.329589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:27.410 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:27.405084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:27.422 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:27.416969 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:37.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:37.479907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:37.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:37.492017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:47.573 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:47.564021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:47.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:47.613604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:57.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:57.713647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:57.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:46:57.724939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:07.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:07.791685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:07.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:07.803595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:17.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:17.880783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:17.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:17.895248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:27.979 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:27.973324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:27.991 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:27.986505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:38.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:38.097291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:38.117 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:38.112420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:48.243 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:48.237557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:48.260 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:48.254973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:58.369 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:58.363749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:58.381 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:47:58.375190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:08.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:08.483267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:08.502 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:08.496350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:18.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:18.589371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:18.608 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:18.602719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:28.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:28.691459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:28.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:28.703477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:38.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:38.832733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:38.849 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:38.843563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:48.948 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:48.941555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:48.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:48.954874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:59.065 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:59.058847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:59.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:48:59.071739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:09.161 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:09.155231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:09.177 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:09.170486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:19.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:19.229128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:19.254 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:19.243121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:29.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:29.305176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:29.327 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:29.319026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:33.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:33.241071 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T02:49:34.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:34.050225 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T02:49:34.037+0000 7f1223009380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T02:49:34.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T02:49:34.037+0000 7f1223009380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T02:49:34.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T02:49:34.037+0000 7f1223009380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T02:49:34.058 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T02:49:39.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:39.461243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:39.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:39.471558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:49.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:49.626298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:49.646 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:49.639878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:59.726 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:59.719703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:59.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:49:59.789637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:09.894 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:09.886888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:09.905 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:09.898732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:19.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:19.956319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:19.980 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:19.973368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:30.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:30.109560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:30.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:30.122862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:40.221 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:40.213589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:40.374 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:40.366938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:50.526 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:50.518874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:50.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:50:50.530919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:00.608 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:00.600709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:00.620 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:00.612874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:10.786 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:10.779473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:10.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:10.791938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:20.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:20.860789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:20.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:20.872901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:30.944 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:30.937277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:30.959 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:30.951752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:41.027 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:41.019392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:41.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:41.034239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:51.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:51.092679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:51.113 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:51:51.105193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:01.177 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:01.169197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:01.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:01.178940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:11.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:11.242734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:11.264 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:11.256332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:21.373 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:21.365447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:21.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:21.377661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:31.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:31.450061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:31.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:31.462461 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:41.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:41.598451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:41.621 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:41.613684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:51.714 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:51.705821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:51.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:52:51.716336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:01.885 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:01.877600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:01.896 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:01.888681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:12.012 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:12.003645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:12.026 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:12.018226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:22.138 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:22.129584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:22.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:22.141779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:32.212 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:32.203843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:32.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:32.214739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:42.288 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:42.280232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:42.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:42.294817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:52.353 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:52.344586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:52.365 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:53:52.356524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:02.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:02.422778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:02.498 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:02.489978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:12.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:12.555146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:12.577 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:12.568629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:22.692 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:22.683321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:22.704 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:22.696137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:32.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:32.760116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:32.781 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:32.773355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:42.846 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:42.837616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:42.859 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:42.850204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:52.955 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:52.946567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:52.969 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:54:52.961188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:03.058 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:03.049776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:03.070 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:03.061899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:13.204 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:13.194617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:13.215 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:13.206832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:23.277 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:23.268668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:23.290 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:23.281184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:33.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:33.351524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:33.452 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:33.443613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:43.555 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:43.546557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:43.571 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:43.562166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:53.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:53.656406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:53.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:55:53.669352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:03.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:03.744973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:03.769 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:03.759922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:13.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:13.870741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:13.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:13.884269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:23.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:23.955845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:23.975 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:23.965931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:34.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:34.040008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:34.059 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:34.050640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:44.176 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:44.166534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:44.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:44.179795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:54.249 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:54.239927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:54.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:56:54.289326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:04.378 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:04.367969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:04.441 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:04.431484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:14.541 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:14.532454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:14.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:14.544420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:24.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:24.664684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:24.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:24.677460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:34.792 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:34.783159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:34.853 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:34.844446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:44.937 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:44.928054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:44.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:44.986291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:55.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:55.073048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:55.093 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:57:55.084743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:05.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:05.148487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:05.171 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:05.162223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:15.270 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:15.261208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:15.280 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:15.271790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:25.464 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:25.455189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:25.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:25.468324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:35.569 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:35.561003 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:35.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:35.575239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:45.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:45.688243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:45.707 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:45.698941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:55.770 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:55.761230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:55.782 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:58:55.773789 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:05.963 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:05.954585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:05.976 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:05.967133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:16.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:16.024385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:16.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:16.038013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:26.114 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:26.105629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:26.127 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:26.118654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:36.192 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:36.183966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:36.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:36.197070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:46.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:46.262956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:46.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:46.274266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:56.344 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:56.336228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:56.358 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 02:59:56.349795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:06.477 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:06.468420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:06.491 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:06.482781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:16.603 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:16.594926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:16.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:16.609609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:26.676 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:26.668230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:26.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:26.728321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:36.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:36.794692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:36.814 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:36.806658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:46.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:46.875391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:46.896 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:46.887797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:56.986 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:56.977680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:56.998 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:00:56.989683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:07.069 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:07.061023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:07.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:07.114235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:17.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:17.180215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:17.201 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:17.192968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:27.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:27.386070 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:27.406 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:27.398630 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:37.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:37.472136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:37.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:37.485037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:47.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:47.580467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:47.602 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:47.594449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:57.668 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:57.660769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:57.680 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:01:57.672717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:07.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:07.735711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:07.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:07.748697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:17.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:17.808277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:17.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:17.821384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:27.926 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:27.918498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:27.940 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:27.932755 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:38.053 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:38.045067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:38.066 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:38.058243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:48.134 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:48.126117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:48.146 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:48.138518 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:58.267 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:58.259935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:58.282 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:02:58.274248 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:08.419 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:08.411031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:08.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:08.423523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:18.528 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:18.520642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:18.545 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:18.537441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:28.742 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:28.734393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:28.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:28.747044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:38.860 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:38.852753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:38.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:38.862652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:49.008 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:49.000584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:49.024 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:49.017290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:59.148 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:59.141460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:59.162 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:03:59.154858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:09.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:09.302942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:09.322 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:09.314865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:19.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:19.450855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:19.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:19.463438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:29.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:29.525385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:29.599 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:29.591736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:39.677 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:39.670548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:39.690 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:39.683435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:49.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:49.788914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:49.807 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:49.800689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:59.879 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:59.871838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:59.891 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:04:59.884544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:09.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:09.954110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:09.973 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:09.966264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:20.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:20.030794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:20.050 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:20.043834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:30.152 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:30.145773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:30.163 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:30.156776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:40.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:40.219000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:40.285 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:40.278411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:50.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:50.385195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:50.404 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:05:50.397746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:00.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:00.463321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:00.484 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:00.476406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:10.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:10.571419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:10.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:10.583146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:14.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:14.070705 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T03:06:14.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:14.728677 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T03:06:14.712+0000 7f37cf54c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T03:06:14.750 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T03:06:14.712+0000 7f37cf54c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T03:06:14.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T03:06:14.712+0000 7f37cf54c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T03:06:14.751 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T03:06:20.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:20.644084 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:20.664 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:20.658271 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:30.732 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:30.725348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:30.783 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:30.776689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:40.914 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:40.907450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:40.927 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:40.920300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:51.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:51.011339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:51.034 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:06:51.027545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:01.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:01.109023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:01.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:01.122420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:11.230 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:11.223477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:11.246 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:11.239053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:21.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:21.325522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:21.345 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:21.338674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:31.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:31.437289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:31.460 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:31.452596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:41.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:41.576239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:41.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:41.588689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:51.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:51.656693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:51.676 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:07:51.669360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:01.746 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:01.739526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:01.757 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:01.750826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:11.813 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:11.805904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:11.825 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:11.818688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:21.951 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:21.944009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:21.964 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:21.957305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:32.032 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:32.025175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:32.047 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:32.041035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:42.110 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:42.103431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:42.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:42.115543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:52.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:52.216911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:52.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:08:52.229890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:02.308 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:02.301609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:02.321 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:02.314333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:12.439 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:12.432422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:12.453 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:12.446720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:22.523 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:22.516574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:22.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:22.588541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:32.705 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:32.698772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:32.720 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:32.712966 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:42.827 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:42.819254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:42.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:42.832317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:52.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:52.896332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:52.949 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:09:52.929730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:03.027 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:03.021024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:03.043 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:03.036653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:13.106 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:13.099444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:13.119 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:13.112599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:23.220 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:23.213950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:23.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:23.227307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:33.302 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:33.295537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:33.321 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:33.315050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:43.442 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:43.435463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:43.455 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:43.448838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:53.539 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:53.532980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:53.554 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:10:53.545477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:03.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:03.611802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:03.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:03.662610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:13.772 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:13.765287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:13.817 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:13.793689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:23.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:23.888499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:23.906 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:23.900300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:34.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:34.011815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:34.030 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:34.024080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:44.124 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:44.117764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:44.694 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:44.130437 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:54.193 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:54.186664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:54.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:11:54.236916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:04.300 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:04.294082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:04.312 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:04.305946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:14.421 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:14.415018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:14.435 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:14.428457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:24.508 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:24.502186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:24.521 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:24.514851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:34.584 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:34.578082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:34.641 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:34.635208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:44.748 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:44.741217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:44.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:44.753620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:54.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:54.852038 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:54.873 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:12:54.866208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:04.943 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:04.936556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:05.039 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:05.032748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:15.121 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:15.114458 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:15.131 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:15.124848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:25.214 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:25.208129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:25.227 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:25.220666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:35.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:35.328406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:35.347 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:35.340381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:45.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:45.463167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:45.484 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:45.477169 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:55.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:55.588426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:55.604 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:13:55.598337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:05.716 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:05.709162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:05.729 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:05.722308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:15.791 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:15.784368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:15.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:15.796989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:25.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:25.862510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:25.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:25.873472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:35.947 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:35.940643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:35.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:35.953384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:46.065 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:46.057111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:46.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:46.069585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:56.181 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:56.175128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:56.202 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:14:56.195721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:06.309 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:06.302618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:06.322 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:06.315763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:16.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:16.378520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:16.448 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:16.441595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:26.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:26.545429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:26.566 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:26.559807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:36.685 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:36.679275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:36.702 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:36.696015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:46.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:46.831274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:46.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:46.844627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:56.919 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:56.912805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:56.929 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:15:56.921756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:07.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:06.995396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:07.074 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:07.067591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:17.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:17.150253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:17.210 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:17.203620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:27.346 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:27.339712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:27.359 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:27.352688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:37.467 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:37.460686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:37.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:37.474552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:47.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:47.583299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:47.602 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:47.595611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:57.690 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:57.678445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:57.699 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:16:57.692668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:07.795 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:07.788830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:07.808 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:07.801693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:17.869 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:17.863252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:17.904 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:17.897745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:27.993 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:27.986167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:28.005 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:27.998850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:38.088 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:38.081870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:38.101 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:38.094993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:48.165 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:48.159079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:48.183 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:48.177341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:58.297 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:58.291295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:58.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:17:58.303994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:08.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:08.397178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:08.422 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:08.416026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:18.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:18.550970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:18.571 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:18.564744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:28.672 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:28.665271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:28.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:28.677711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:38.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:38.796798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:38.816 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:38.809612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:48.918 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:48.912164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:48.930 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:48.923962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:59.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:59.077167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:59.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:18:59.090480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:09.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:09.198435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:09.239 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:09.232803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:19.316 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:19.309661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:19.328 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:19.321590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:29.430 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:29.423436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:29.442 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:29.435352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:39.552 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:39.546138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:39.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:39.557620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:49.662 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:49.655483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:49.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:49.667814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:59.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:59.734412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:59.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:19:59.747282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:09.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:09.823121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:09.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:09.837466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:19.939 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:19.933252 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:19.954 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:19.948014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:30.070 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:30.063939 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:30.084 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:30.077828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:40.206 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:40.199339 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:40.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:40.216923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:50.324 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:50.317947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:50.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:20:50.330007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:00.411 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:00.404959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:00.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:00.416685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:10.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:10.479319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:10.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:10.491481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:20.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:20.559434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:20.575 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:20.568865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:30.654 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:30.647713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:30.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:30.660521 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:40.740 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:40.733288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:40.752 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:40.745786 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:50.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:50.831596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:50.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:21:50.844527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:00.919 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:00.912317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:00.931 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:00.924863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:10.994 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:10.987116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:11.005 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:10.998833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:21.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:21.121769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:21.141 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:21.134561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:31.264 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:31.257326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:31.275 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:31.268233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:41.385 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:41.377713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:41.418 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:41.411322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:51.518 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:51.510898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:51.531 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:51.523079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:54.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:54.738215 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T03:22:55.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:22:55.423421 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T03:22:55.409+0000 7f637690b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T03:22:55.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T03:22:55.409+0000 7f637690b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T03:22:55.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T03:22:55.409+0000 7f637690b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T03:22:55.432 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T03:23:01.603 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:01.595856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:01.621 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:01.613880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:11.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:11.689625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:11.712 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:11.704858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:21.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:21.778149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:21.852 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:21.844865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:32.004 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:31.996543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:32.018 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:32.010463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:42.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:42.108545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:42.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:42.121706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:52.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:52.225210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:52.244 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:23:52.237145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:02.323 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:02.315146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:02.338 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:02.331108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:12.436 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:12.427731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:12.449 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:12.441660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:22.572 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:22.563671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:22.583 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:22.575091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:32.660 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:32.651891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:32.674 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:32.666083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:42.744 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:42.735891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:42.768 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:42.760447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:52.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:52.862424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:52.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:24:52.876556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:02.982 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:02.974258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:02.993 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:02.985589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:13.129 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:13.120337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:13.141 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:13.133051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:23.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:23.247297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:23.285 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:23.277233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:33.398 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:33.389236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:33.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:33.398921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:43.491 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:43.482559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:43.501 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:43.493149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:53.605 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:53.596793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:53.618 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:25:53.609575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:03.721 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:03.712901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:03.773 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:03.763680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:13.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:13.824332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:13.845 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:13.836833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:23.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:23.951537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:23.974 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:23.965323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:34.042 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:34.032841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:34.055 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:34.045963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:44.153 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:44.144015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:44.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:44.156913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:54.233 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:54.224387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:54.245 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:26:54.235651 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:04.313 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:04.303540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:04.325 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:04.316123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:14.400 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:14.390707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:14.413 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:14.403943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:24.486 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:24.476681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:24.557 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:24.547897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:34.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:34.657593 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:34.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:34.670136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:44.756 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:44.746113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:44.816 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:44.806905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:54.890 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:54.880698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:54.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:27:54.891358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:04.976 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:04.966548 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:04.990 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:04.980087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:15.130 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:15.120594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:15.142 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:15.132884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:25.211 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:25.201358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:25.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:25.216990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:35.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:35.279251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:35.378 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:35.367657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:45.468 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:45.458798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:45.481 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:45.471826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:55.574 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:55.564130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:55.587 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:28:55.577318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:05.656 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:05.646629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:05.670 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:05.660375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:15.776 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:15.766346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:15.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:15.780643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:25.952 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:25.942228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:25.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:25.951434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:36.110 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:36.100314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:36.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:36.113386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:46.382 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:46.372209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:46.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:46.381961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:56.667 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:56.656943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:56.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:29:56.670902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:06.857 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:06.846278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:06.868 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:06.857996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:16.994 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:16.983326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:17.004 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:16.994149 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:27.086 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:27.075630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:27.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:27.089117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:37.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:37.145065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:37.176 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:37.165570 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:47.242 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:47.231879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:47.256 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:47.245610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:57.323 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:57.312231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:57.338 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:30:57.327923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:07.529 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:07.517293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:07.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:07.547901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:17.745 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:17.734220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:17.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:17.745060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:27.861 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:27.850299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:27.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:27.863426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:38.007 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:37.996281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:38.020 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:38.009089 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:48.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:48.081148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:48.104 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:48.093744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:58.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:58.164765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:58.189 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:31:58.177172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:08.258 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:08.246675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:08.271 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:08.259963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:18.379 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:18.368390 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:18.392 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:18.381526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:28.479 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:28.468364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:28.493 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:28.482582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:38.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:38.554423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:38.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:38.568006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:48.673 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:48.661975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:48.686 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:48.674896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:58.749 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:58.738159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:58.762 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:32:58.750808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:08.859 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:08.847637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:08.871 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:08.859883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:18.939 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:18.928151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:18.953 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:18.941704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:29.028 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:29.016516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:29.041 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:29.030328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:39.103 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:39.091568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:39.116 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:39.104927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:49.250 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:49.238824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:49.263 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:49.251702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:59.389 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:59.377796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:59.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:33:59.391818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:09.507 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:09.494644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:09.529 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:09.517750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:19.621 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:19.609215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:19.632 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:19.620941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:29.723 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:29.711381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:29.737 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:29.724888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:39.838 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:39.826100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:39.851 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:39.837067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:49.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:49.896554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:49.920 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:49.908938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:00.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:34:59.991857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:00.016 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:00.005111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:10.102 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:10.090631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:10.114 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:10.102803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:20.175 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:20.160814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:20.184 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:20.172063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:30.252 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:30.240587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:30.264 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:30.252639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:40.360 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:40.348205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:40.371 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:40.359802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:50.450 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:50.437897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:50.461 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:35:50.449736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:00.562 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:00.550380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:00.574 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:00.562127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:10.634 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:10.622076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:10.646 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:10.634022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:20.829 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:20.817613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:20.842 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:20.830709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:30.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:30.890618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:30.944 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:30.932513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:41.060 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:41.048028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:41.072 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:41.059933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:51.132 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:51.119989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:51.142 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:36:51.130592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:01.254 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:01.241780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:01.266 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:01.254091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:11.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:11.324126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:11.349 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:11.337128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:21.453 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:21.441154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:21.465 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:21.453141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:31.614 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:31.601387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:31.694 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:31.615615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:41.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:41.684511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:41.713 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:41.701527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:51.780 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:51.767759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:51.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:37:51.778593 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:01.882 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:01.870384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:01.894 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:01.882282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:11.957 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:11.945019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:11.972 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:11.960046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:22.068 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:22.055653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:22.079 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:22.067431 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:32.150 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:32.137326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:32.163 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:32.150718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:42.246 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:42.233460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:42.259 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:42.246315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:52.359 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:52.346192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:52.374 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:38:52.360861 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:02.441 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:02.428108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:02.451 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:02.438582 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:12.515 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:12.502540 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:12.527 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:12.514355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:22.593 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:22.580549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:22.606 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:22.593252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:32.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:32.648721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:32.673 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:32.659335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:35.478 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:35.465627 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T03:39:36.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:36.143481 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T03:39:36.135+0000 7f05b5803380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T03:39:36.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T03:39:36.135+0000 7f05b5803380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T03:39:36.157 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T03:39:36.135+0000 7f05b5803380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T03:39:36.158 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T03:39:42.777 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:42.764753 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:42.789 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:42.777042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:52.906 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:52.893456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:52.917 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:39:52.904246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:03.031 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:03.018413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:03.043 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:03.030629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:13.111 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:13.098108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:13.182 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:13.169143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:23.247 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:23.234017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:23.260 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:23.247616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:33.332 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:33.319231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:33.345 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:33.332712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:43.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:43.485607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:43.511 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:43.498219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:53.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:53.616113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:53.645 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:40:53.632427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:03.814 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:03.800751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:03.847 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:03.833397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:13.938 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:13.924417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:13.949 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:13.936219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:24.026 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:24.012738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:24.037 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:24.024863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:34.165 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:34.151717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:34.177 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:34.164030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:44.290 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:44.277127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:44.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:44.292035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:54.408 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:54.395203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:54.419 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:41:54.406852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:04.487 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:04.474568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:04.500 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:04.487301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:14.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:14.616159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:14.643 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:14.629987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:24.742 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:24.728528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:24.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:24.738866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:34.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:34.860831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:34.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:34.875266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:44.950 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:44.937212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:44.964 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:44.951911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:55.079 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:55.065594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:55.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:42:55.077185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:05.188 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:05.175350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:05.202 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:05.189881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:15.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:15.305504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:15.330 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:15.317254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:25.407 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:25.393935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:25.469 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:25.456905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:35.576 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:35.562959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:35.589 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:35.576446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:45.665 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:45.652637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:45.676 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:45.663074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:55.743 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:55.730317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:55.755 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:43:55.742308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:05.811 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:05.797915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:05.824 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:05.811575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:15.920 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:15.906710 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:15.932 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:15.919298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:26.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:25.988962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:26.064 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:26.050647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:36.291 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:36.277866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:36.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:36.292284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:46.410 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:46.397575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:46.423 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:46.411030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:56.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:56.486528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:56.515 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:44:56.502734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:06.623 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:06.610752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:06.636 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:06.623374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:16.700 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:16.687811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:16.717 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:16.704780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:26.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:26.772715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:26.796 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:26.783368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:36.858 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:36.845367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:36.870 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:36.857838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:46.937 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:46.924343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:46.951 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:46.937184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:57.020 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:57.007482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:57.739 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:45:57.020390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:07.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:07.120960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:07.148 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:07.135505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:17.254 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:17.241119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:17.266 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:17.253516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:27.335 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:27.322764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:27.347 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:27.334403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:37.412 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:37.399435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:37.424 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:37.411691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:47.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:47.551205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:47.577 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:47.564077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:57.669 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:57.656760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:57.682 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:46:57.668959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:07.797 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:07.784983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:07.810 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:07.797388 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:17.885 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:17.871998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:17.898 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:17.885375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:27.965 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:27.952861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:27.980 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:27.967734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:38.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:38.068830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:38.094 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:38.082174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:48.176 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:48.163707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:48.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:48.173626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:58.279 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:58.266149 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:58.290 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:47:58.277370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:08.371 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:08.358985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:08.383 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:08.371030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:18.496 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:18.483941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:18.508 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:18.495594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:28.638 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:28.626128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:28.653 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:28.641255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:38.799 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:38.786431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:38.810 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:38.797673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:48.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:48.871603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:48.896 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:48.883603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:58.967 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:58.954338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:58.980 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:48:58.967763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:09.192 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:09.178587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:09.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:09.192843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:19.297 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:19.284641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:19.359 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:19.346372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:29.444 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:29.431061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:29.453 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:29.440911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:39.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:39.554257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:39.580 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:39.567514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:49.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:49.671850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:49.700 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:49.687189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:59.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:59.748603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:59.816 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:49:59.803618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:09.936 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:09.923668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:09.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:09.934372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:20.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:20.033236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:20.057 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:20.045260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:30.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:30.121158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:30.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:30.132826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:40.249 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:40.236091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:40.261 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:40.248600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:50.326 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:50.313759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:50.435 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:50:50.327682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:00.439 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:00.427241 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:00.452 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:00.440219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:10.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:10.552860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:10.576 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:10.563349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:20.666 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:20.653803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:20.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:20.666354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:30.785 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:30.773177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:30.798 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:30.785658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:40.895 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:40.883202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:40.907 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:40.895175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:50.981 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:50.968256 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:50.994 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:51:50.981001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:01.101 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:01.089135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:01.114 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:01.102094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:11.206 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:11.193729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:11.222 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:11.209982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:21.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:21.304391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:21.328 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:21.316585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:31.439 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:31.426855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:31.450 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:31.438661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:41.543 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:41.530599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:41.558 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:41.545949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:51.661 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:51.648759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:51.673 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:52:51.661386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:01.751 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:01.738871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:01.760 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:01.748808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:11.888 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:11.875840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:11.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:11.888060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:22.005 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:21.992492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:22.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:22.007227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:32.082 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:32.070150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:32.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:32.087937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:42.195 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:42.182699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:42.209 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:42.195440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:52.276 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:52.263906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:52.289 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:53:52.277065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:02.359 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:02.346566 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:02.372 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:02.359734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:12.445 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:12.432616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:12.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:12.445807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:22.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:22.575368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:22.600 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:22.587767 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:32.712 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:32.699510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:32.724 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:32.712240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:42.790 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:42.777854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:42.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:42.791244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:52.911 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:52.898774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:52.922 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:54:52.910052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:02.984 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:02.971916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:02.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:02.983353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:13.109 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:13.096679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:13.122 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:13.110205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:23.183 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:23.170778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:23.196 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:23.184822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:33.305 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:33.292559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:33.317 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:33.304699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:43.427 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:43.414360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:43.440 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:43.427883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:53.538 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:53.526204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:53.550 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:55:53.538047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:03.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:03.630955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:03.655 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:03.643288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:13.761 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:13.750273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:13.919 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:13.764049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:16.198 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:16.187070 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T03:56:16.883 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:16.871966 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T03:56:16.861+0000 7fdb087a5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T03:56:16.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T03:56:16.861+0000 7fdb087a5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T03:56:16.884 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T03:56:16.861+0000 7fdb087a5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T03:56:16.884 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T03:56:23.874 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:23.863435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:23.886 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:23.875245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:33.992 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:33.981355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:34.003 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:33.992790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:44.118 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:44.107511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:44.675 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:44.118515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:54.205 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:54.195052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:54.284 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:56:54.274330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:04.396 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:04.386340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:04.409 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:04.399699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:14.476 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:14.466961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:14.528 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:14.518845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:24.627 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:24.617559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:24.640 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:24.631406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:34.706 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:34.697661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:34.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:34.710151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:44.788 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:44.779199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:44.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:44.792860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:54.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:54.914260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:54.934 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:57:54.926124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:05.002 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:04.994102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:05.014 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:05.006315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:15.136 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:15.128471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:15.151 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:15.142422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:25.218 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:25.210676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:25.231 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:25.222641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:35.310 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:35.302319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:35.323 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:35.315813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:45.431 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:45.423790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:45.446 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:45.439137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:55.540 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:55.533326 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:55.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:58:55.546301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:05.663 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:05.655909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:05.678 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:05.671525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:15.788 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:15.781767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:15.802 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:15.796196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:25.937 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:25.930335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:25.949 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:25.942359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:36.046 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:36.039338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:36.062 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:36.055726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:46.135 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:46.128703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:46.149 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:46.143252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:56.289 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:56.283454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:56.301 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 03:59:56.295063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:06.363 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:06.357788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:06.435 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:06.429441 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:16.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:16.493911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:16.512 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:16.507275 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:26.574 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:26.568486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:26.636 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:26.631376 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:36.733 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:36.727761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:36.747 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:36.741831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:46.834 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:46.829501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:46.846 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:46.841824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:56.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:56.956404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:56.983 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:00:56.978791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:07.112 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:07.107088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:07.126 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:07.121457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:17.248 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:17.243694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:17.261 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:17.256923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:27.398 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:27.393902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:27.411 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:27.407305 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:37.492 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:37.487860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:37.503 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:37.499905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:47.622 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:47.618074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:47.633 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:47.629877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:57.697 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:57.693086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:57.710 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:01:57.706408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:07.826 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:07.822826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:07.842 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:07.838399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:17.903 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:17.899792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:17.970 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:17.967113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:28.077 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:28.073195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:28.091 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:28.087990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:38.155 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:38.150751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:38.168 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:38.165148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:48.236 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:48.233348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:48.251 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:48.248137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:58.319 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:58.316846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:58.396 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:02:58.393129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:08.485 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:08.482195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:08.497 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:08.495420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:18.595 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:18.592017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:18.607 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:18.605452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:28.721 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:28.718448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:28.734 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:28.731587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:38.809 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:38.807557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:38.856 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:38.854920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:48.919 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:48.916874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:48.942 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:48.934757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:59.006 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:59.004647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:59.019 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:03:59.017807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:09.142 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:09.140243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:09.154 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:09.152802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:19.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:19.221810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:19.235 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:19.234143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:29.303 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:29.301804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:29.374 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:29.372718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:39.489 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:39.487618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:39.503 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:39.501668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:49.576 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:49.575628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:49.588 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:49.586821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:59.657 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:59.655907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:59.670 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:04:59.669691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:09.781 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:09.780408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:09.791 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:09.790894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:19.900 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:19.899331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:19.914 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:19.912662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:30.013 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:30.012778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:30.023 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:30.022940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:40.126 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:40.125776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:40.139 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:40.138536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:50.311 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:50.310935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:50.323 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:05:50.322959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:00.422 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:00.421825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:00.434 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:00.433379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:10.540 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:10.539745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:10.559 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:10.549916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:20.642 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:20.642671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:20.655 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:20.655690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:30.729 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:30.729688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:30.742 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:30.742597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:40.864 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:40.864082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:40.877 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:40.877293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:50.981 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:50.981265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:50.995 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:06:50.995917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:01.132 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:01.132625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:01.145 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:01.145728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:11.211 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:11.211864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:11.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:11.223494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:21.336 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:21.336654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:21.348 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:21.349046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:31.416 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:31.416647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:31.428 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:31.429270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:41.532 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:41.532803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:41.549 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:41.547987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:51.641 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:51.641976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:51.654 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:07:51.655514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:01.766 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:01.767334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:02.021 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:01.779693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:11.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:11.844305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:11.852 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:11.854103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:21.948 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:21.949322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:21.961 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:21.962703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:32.049 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:32.050989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:32.099 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:32.100371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:42.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:42.191221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:42.201 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:42.202312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:52.318 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:52.319901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:52.330 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:08:52.332095 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:02.422 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:02.423756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:02.435 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:02.437164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:12.519 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:12.520361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:12.532 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:12.534025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:22.639 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:22.640444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:22.651 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:22.653364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:32.719 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:32.720918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:32.731 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:32.733228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:42.792 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:42.794371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:42.843 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:42.845439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:52.909 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:52.911313 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:52.921 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:09:52.923855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:03.043 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:03.045804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:03.055 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:03.057357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:13.166 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:13.168717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:13.183 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:13.184381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:23.249 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:23.251432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:23.264 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:23.266705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:33.367 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:33.370033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:33.381 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:33.383713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:43.483 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:43.485293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:43.496 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:43.498447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:53.564 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:53.567001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:53.579 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:10:53.581519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:03.659 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:03.662263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:03.723 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:03.726003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:13.845 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:13.847646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:13.857 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:13.859604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:23.974 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:23.977507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:23.988 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:23.991102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:34.101 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:34.103894 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:34.113 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:34.116107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:44.210 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:44.213011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:44.223 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:44.226133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:54.320 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:54.323764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:54.331 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:11:54.334174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:04.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:04.397589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:04.404 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:04.407578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:14.470 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:14.473267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:14.483 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:14.486726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:24.631 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:24.634760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:24.645 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:24.648228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:34.723 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:34.726993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:34.736 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:34.739959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:44.835 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:44.838427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:44.848 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:44.850881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:54.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:54.949395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:54.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:54.963547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:56.875 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:56.878698 I | op-mon: parsing mon endpoints: a=172.21.15.19:6789,b=172.21.15.18:6789,c=172.21.15.59:6789 2022-03-06T04:12:57.536 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:12:57.538620 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T04:12:57.527+0000 7fc8c577e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T04:12:57.536 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T04:12:57.527+0000 7fc8c577e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T04:12:57.537 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06T04:12:57.527+0000 7fc8c577e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T04:12:57.537 INFO:tasks.rook.operator.smithi018.stdout:couldn't init storage provider. : exit status 5 2022-03-06T04:13:05.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:05.039835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:05.096 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:05.099407 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:15.190 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:15.194115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:15.203 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:15.206592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:25.269 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:25.273050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:25.283 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:25.286510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:35.350 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:35.353822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:35.364 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:35.367290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:45.448 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:45.452464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:45.458 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:45.462616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:55.567 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:55.571493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:55.584 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:13:55.587798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:05.671 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:05.675747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:05.683 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:05.686987 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:15.754 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:15.758034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:15.768 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:15.772924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:25.833 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:25.837175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:25.848 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:25.852692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:35.946 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:35.950798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:35.960 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:35.964680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:46.081 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:46.085861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:46.095 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:46.099703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:56.209 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:56.213142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:56.222 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:14:56.226542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:06.354 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:06.358947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:06.368 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:06.372656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:16.553 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:16.557848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:16.565 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:16.570100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:26.637 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:26.641805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:26.679 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:26.683601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:36.789 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:36.793798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:36.801 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:36.806731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:46.876 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:46.881351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:46.891 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:46.896512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:57.010 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:57.014926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:57.022 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:15:57.027200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:07.083 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:07.087952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:07.100 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:07.104799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:17.226 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:17.230820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:17.237 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:17.242644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:27.394 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:27.398716 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:27.410 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:27.415700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:37.499 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:37.504358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:37.563 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:37.567901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:47.617 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:47.621850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:47.629 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:47.634500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:57.730 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:57.735571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:57.744 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:16:57.749499 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:07.835 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:07.839964 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:07.847 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:07.852314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:17.953 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:17.958435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:17.966 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:17.972039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:28.043 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:28.048148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:28.056 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:28.061428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:38.123 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:38.129224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:38.135 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:38.140896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:48.314 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:48.319437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:48.327 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:48.333140 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:58.464 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:58.469157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:58.476 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:17:58.481279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:08.596 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:08.601148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:08.608 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:08.614058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:18.684 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:18.689370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:18.696 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:18.701479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:28.803 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:28.808655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:28.814 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:28.820444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:38.910 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:38.915889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:38.923 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:38.929051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:49.036 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:49.041712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:49.048 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:49.053605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:59.120 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:59.125669 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:59.133 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:18:59.139276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:09.228 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:09.234408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:09.242 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:09.247943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:19.372 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:19.377906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:19.384 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:19.390683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:29.480 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:29.486257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:29.511 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:29.517163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:39.601 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:39.607503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:39.681 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:39.621486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:49.690 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:49.696022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:49.703 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:49.709531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:59.804 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:59.810243 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:59.816 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:19:59.822507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:09.880 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:09.886684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:09.893 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:09.899815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:19.997 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:20.003933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:20.009 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:20.015337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:30.076 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:30.083067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:30.092 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:30.097864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:40.174 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:40.180446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:40.186 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:40.192374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:50.270 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:50.276993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:20:50.285 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:20:50.291383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:21:00.403 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:21:00.409805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:21:00.415 INFO:tasks.rook.operator.smithi018.stdout:2022-03-06 04:21:00.421863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T06:41:55.720 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-06T06:41:55.749 DEBUG:teuthology.task.console_log:Killing console logger for smithi018 2022-03-06T06:41:55.750 DEBUG:teuthology.task.console_log:Killing console logger for smithi019 2022-03-06T06:41:55.751 DEBUG:teuthology.task.console_log:Killing console logger for smithi059 2022-03-06T06:41:55.751 DEBUG:teuthology.exit:Finished running handlers