2022-03-06T00:08:52.076 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-06T00:08:52.079 DEBUG:teuthology.run:Teuthology command: teuthology --name ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi --owner scheduled_ozeneva@teuthology --archive /home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723214 --verbose --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} -- /home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723214/orig.config.yaml 2022-03-06T00:08:52.130 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-06T00:08:52.201 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723214 branch: wip-omri-tracing-compiled description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} email: null first_in_suite: false job_id: '6723214' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-omri-tracing-compiled ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-omri-tracing-compiled sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 owner: scheduled_ozeneva@teuthology priority: 50 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 sleep_before_teardown: 0 suite: rados suite_branch: wip-omri-tracing-compiled suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 targets: smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKlTEEtze8Cgy+9gzve63O5oamGGF4VZNCo1clB/pYJyGpwRJzvyqESaA3f8hahin0UBHfXm4HWuy9xkqrmGNxxruW7oyoMdCwemD+ODifuNQv6SUtr3BZJjvDRu+nIqwbmJs3/gfyav17QPchjZvzL0atG7VBGKOz7bhGI9uPqm3joxMPqCh8PZnpo/eihXJjcylu6O2Jn7byAVh777IjxYTC9ZBLiDfiPfTHg939N1CKboP/ZznAh0s/VDWhUNpbzj6KSmJ54gekk/slXb8JJwsOerudOkn7+qEuSzW7ErKPK0qhs3iju5X+UYuUZmlUrkCRae+3OTbIYRcsk5+3aEbIOqDsgf3wTD61zkcrswNne+TE0VuKAWzuqp9vZozDb6WBmAMwAGffZJXPdQnuiIM0wQ0CO4Q2SDoXjmss7tZtawIDOEH4NgX6mQhBJwFgQ/v5GUjdtoB4x8D+oFeyq0tfVKg+bra9z5bdDal+PfLVm2GAfNseed8dOQc5lZ8= smithi017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJACsZeNloei8tPRAohyyTsQDgEp9+bj2uvn6fdxGEwmqEat81cdS27JHPfw4Emhj9lbQj+worPU2sYK3LyVVVUP6S7/BX0RoCtcL6ldc6FDiS7zvnGFrMrU5u4ubp69a0SNgGo0KY7xafUwnUSx/brpxng8jkTPYNbb0RXgOpHQFI2oB5zqGxRtlHkBnhkuCTrv02BUh8RAuCOUCXkrYnzgO3z1vImkvPolaC6EWVvEmNtizJRBDJryr3/J6vQFMVepkoZiNG7a1SdWpADqjoOBXGF9E+hJP0GQy3V5JaFtN2NHy4rnnwG0viWZyOs0Q7gjGKHzheQB7sXaF8sZai0Y7MQ4IRjpxvr1K2H4kZLL/DzANwK+jnuYxP1umXqkkHPduM6zKRuKZLPtObs7IwSOXmEg9uiLPijmem3LZz8hNJmtBYe4ygtEkAbXAe1bE5GMkfgVbGXgX570x6Bfo2Z4U7sv72fPmp6c+hjTzCjYMDidnoPFQrUUQTaeOFT3U= smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC96QBcxk/YY86Lk9wsINK62XoFGDHKEI8/G+jAJyHpywZJ1AWueiHG6ivdty9p4libcYoXnhZHJNMIHjYu9sfdkMo3GIQDSWTTMgGcfRT/TsHFPB9gZMmhcOabNWEUjgtvZocu4KKYduVVeNFws0NZZZa0gOF53PtB3TdBDzsXMF+RqVAi2FsJi/6pt3WbNklz74kxLEJtURJIFMjJBBlyMwC90lSDfLfN8Jq3qLG5gaF2MvAz7urKK7jdkDL+t4o7nkeo2mknmythMkaz7uH7W6ArEfjL3co7fMYmpftNYwOZ+nvTin/qHbF1Xu4dYmLD4AQF1AflDvRKowP9WqwY/8bLO3U2v4azCbGkZe8FIvjwPqs4ck7lJpbjqqRRxmg0adXkL9sPvG5qH5vXSIiiO0lHY+EoEqHrQK12drSrKnXYLrO4tDAP3Wgl0SRsLfKTQqzk+23aNBbze+F+xdiFnM53mL2jcn8e/1Z83GEZQOQIgUQ0mXqJiGgledrmB5s= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-06_00:00:14 tube: smithi user: ozeneva verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-06T00:08:52.201 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa; will attempt to use it 2022-03-06T00:08:52.202 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks 2022-03-06T00:08:52.203 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-06T00:08:52.204 INFO:teuthology.task.internal:Checking packages... 2022-03-06T00:08:52.226 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd0dc2fbbf3f64a2d513e2355b245d0ea9728ff97' 2022-03-06T00:08:52.226 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-06T00:08:52.227 INFO:teuthology.packaging:ref: None 2022-03-06T00:08:52.227 INFO:teuthology.packaging:tag: None 2022-03-06T00:08:52.227 INFO:teuthology.packaging:branch: wip-omri-tracing-compiled 2022-03-06T00:08:52.228 INFO:teuthology.packaging:sha1: d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 2022-03-06T00:08:52.228 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-omri-tracing-compiled 2022-03-06T00:08:52.419 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-10974-gd0dc2fbb-1focal 2022-03-06T00:08:52.420 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-06T00:08:52.439 INFO:teuthology.task.internal:no buildpackages task found 2022-03-06T00:08:52.440 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-06T00:08:52.452 INFO:teuthology.task.internal:Saving configuration 2022-03-06T00:08:52.466 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-06T00:08:52.482 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-06T00:08:52.505 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723214', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-06 00:01:53.128028', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ7Qi+Ulw5CLfaklChpUNPG2k3gHeotnkS2ZtHTqFg3s'} 2022-03-06T00:08:52.527 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723214', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-06 00:01:53.127045', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICqF7DV1cThJkhlBHxw4ELUUzFaLr1cX7fcf5Xf77mMh'} 2022-03-06T00:08:52.548 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ozeneva-2022-03-06_00:00:14-rados-wip-omri-tracing-compiled-distro-basic-smithi/6723214', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-06 00:01:53.125669', 'locked_by': 'scheduled_ozeneva@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMNkHAw+G4B0g0jzqZiS0LR3oBCm5x3+U+0bg7ipOCW/'} 2022-03-06T00:08:52.549 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-06T00:08:52.556 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-06T00:08:52.557 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-06T00:08:52.557 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-06T00:08:52.557 INFO:teuthology.run_tasks:Running task console_log... 2022-03-06T00:08:52.689 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7b89923950>, signals=[15]) 2022-03-06T00:08:52.690 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-06T00:08:52.699 INFO:teuthology.task.internal:Opening connections... 2022-03-06T00:08:52.699 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2022-03-06T00:08:52.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:08:52.764 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2022-03-06T00:08:52.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:08:52.822 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2022-03-06T00:08:52.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:08:52.875 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-06T00:08:52.887 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2022-03-06T00:08:52.897 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2022-03-06T00:08:52.898 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2022-03-06T00:08:52.944 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2022-03-06T00:08:52.944 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-06T00:08:52.945 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2022-03-06T00:08:52.945 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2022-03-06T00:08:52.945 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-06T00:08:52.945 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="20.04" 2022-03-06T00:08:52.945 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-06T00:08:52.946 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-06T00:08:52.946 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-06T00:08:52.946 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-06T00:08:52.946 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=focal 2022-03-06T00:08:52.947 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=focal 2022-03-06T00:08:52.947 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2022-03-06T00:08:53.003 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2022-03-06T00:08:53.011 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2022-03-06T00:08:53.012 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2022-03-06T00:08:53.060 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="Ubuntu" 2022-03-06T00:08:53.061 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-06T00:08:53.061 INFO:teuthology.orchestra.run.smithi017.stdout:ID=ubuntu 2022-03-06T00:08:53.061 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE=debian 2022-03-06T00:08:53.062 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-06T00:08:53.062 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="20.04" 2022-03-06T00:08:53.062 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-06T00:08:53.062 INFO:teuthology.orchestra.run.smithi017.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-06T00:08:53.063 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-06T00:08:53.063 INFO:teuthology.orchestra.run.smithi017.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-06T00:08:53.063 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_CODENAME=focal 2022-03-06T00:08:53.063 INFO:teuthology.orchestra.run.smithi017.stdout:UBUNTU_CODENAME=focal 2022-03-06T00:08:53.064 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2022-03-06T00:08:53.081 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2022-03-06T00:08:53.091 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2022-03-06T00:08:53.092 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2022-03-06T00:08:53.100 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="Ubuntu" 2022-03-06T00:08:53.101 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-06T00:08:53.101 INFO:teuthology.orchestra.run.smithi066.stdout:ID=ubuntu 2022-03-06T00:08:53.102 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE=debian 2022-03-06T00:08:53.102 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-06T00:08:53.103 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="20.04" 2022-03-06T00:08:53.103 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-06T00:08:53.104 INFO:teuthology.orchestra.run.smithi066.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-06T00:08:53.104 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-06T00:08:53.104 INFO:teuthology.orchestra.run.smithi066.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-06T00:08:53.104 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_CODENAME=focal 2022-03-06T00:08:53.105 INFO:teuthology.orchestra.run.smithi066.stdout:UBUNTU_CODENAME=focal 2022-03-06T00:08:53.105 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2022-03-06T00:08:53.123 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-06T00:08:53.138 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-06T00:08:53.149 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-06T00:08:53.150 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2022-03-06T00:08:53.153 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2022-03-06T00:08:53.155 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2022-03-06T00:08:53.161 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-06T00:08:53.170 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-06T00:08:53.171 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2022-03-06T00:08:53.205 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2022-03-06T00:08:53.209 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2022-03-06T00:08:53.232 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-06T00:08:53.314 INFO:teuthology.run_tasks:Running task kernel... 2022-03-06T00:08:53.332 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-06T00:08:53.333 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-06T00:08:53.334 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-06T00:08:53.334 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-06T00:08:53.335 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2022-03-06T00:08:53.337 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2022-03-06T00:08:53.338 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2022-03-06T00:08:53.345 INFO:teuthology.orchestra.run.smithi017.stdout:5.4.0-97-generic 2022-03-06T00:08:53.346 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.4.0-97-generic 2022-03-06T00:08:53.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get clean 2022-03-06T00:08:53.349 INFO:teuthology.orchestra.run.smithi066.stdout:5.4.0-97-generic 2022-03-06T00:08:53.349 INFO:teuthology.orchestra.run.smithi007.stdout:5.4.0-97-generic 2022-03-06T00:08:53.350 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.4.0-97-generic 2022-03-06T00:08:53.351 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt-get clean 2022-03-06T00:08:53.352 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.4.0-97-generic 2022-03-06T00:08:53.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2022-03-06T00:08:53.427 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt-get update 2022-03-06T00:08:53.459 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get update 2022-03-06T00:08:53.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2022-03-06T00:08:53.580 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:08:53.669 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:08:53.695 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:08:53.765 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:08:53.776 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:08:53.777 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:08:53.778 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:08:53.826 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:08:53.848 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:08:53.886 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-06T00:08:53.964 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:08:53.983 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:08:54.028 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-06T00:08:54.032 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,740 B] 2022-03-06T00:08:54.033 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-06T00:08:54.049 INFO:teuthology.orchestra.run.smithi066.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-06T00:08:54.049 INFO:teuthology.orchestra.run.smithi066.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-06T00:08:54.050 INFO:teuthology.orchestra.run.smithi066.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-06T00:08:54.050 INFO:teuthology.orchestra.run.smithi066.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:08:54.066 INFO:teuthology.orchestra.run.smithi066.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-06T00:08:54.080 INFO:teuthology.orchestra.run.smithi066.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-06T00:08:54.081 INFO:teuthology.orchestra.run.smithi066.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-06T00:08:54.099 INFO:teuthology.orchestra.run.smithi066.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-06T00:08:54.100 INFO:teuthology.orchestra.run.smithi066.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-06T00:08:54.101 INFO:teuthology.orchestra.run.smithi066.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-06T00:08:54.101 INFO:teuthology.orchestra.run.smithi066.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-06T00:08:54.128 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-06T00:08:54.131 INFO:teuthology.orchestra.run.smithi066.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:08:54.135 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-06T00:08:54.138 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,740 B] 2022-03-06T00:08:54.139 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-06T00:08:54.160 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-06T00:08:54.161 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-06T00:08:54.162 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-06T00:08:54.163 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:08:54.171 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-06T00:08:54.178 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-06T00:08:54.179 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-06T00:08:54.193 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-06T00:08:54.194 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:08:54.195 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-06T00:08:54.195 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-06T00:08:54.195 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-06T00:08:54.200 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:08:54.262 INFO:teuthology.orchestra.run.smithi066.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-06T00:08:54.264 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:08:54.332 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-06T00:08:54.407 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-06T00:08:54.644 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-06T00:08:54.699 INFO:teuthology.orchestra.run.smithi066.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-06T00:08:54.721 INFO:teuthology.orchestra.run.smithi066.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-06T00:08:54.735 INFO:teuthology.orchestra.run.smithi066.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-06T00:08:54.735 INFO:teuthology.orchestra.run.smithi066.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-06T00:08:54.736 INFO:teuthology.orchestra.run.smithi066.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-06T00:08:54.737 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-06T00:08:54.762 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-06T00:08:54.780 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,740 B] 2022-03-06T00:08:54.780 INFO:teuthology.orchestra.run.smithi017.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-06T00:08:54.809 INFO:teuthology.orchestra.run.smithi066.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-06T00:08:54.813 INFO:teuthology.orchestra.run.smithi066.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-06T00:08:54.813 INFO:teuthology.orchestra.run.smithi066.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB] 2022-03-06T00:08:54.852 INFO:teuthology.orchestra.run.smithi017.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-06T00:08:54.853 INFO:teuthology.orchestra.run.smithi017.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-06T00:08:54.855 INFO:teuthology.orchestra.run.smithi066.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [669 kB] 2022-03-06T00:08:54.855 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-06T00:08:54.861 INFO:teuthology.orchestra.run.smithi017.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-06T00:08:54.862 INFO:teuthology.orchestra.run.smithi017.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-06T00:08:54.883 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-06T00:08:54.884 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-06T00:08:54.892 INFO:teuthology.orchestra.run.smithi017.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-06T00:08:54.893 INFO:teuthology.orchestra.run.smithi066.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-06T00:08:54.903 INFO:teuthology.orchestra.run.smithi066.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-06T00:08:54.905 INFO:teuthology.orchestra.run.smithi017.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:08:54.911 INFO:teuthology.orchestra.run.smithi017.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-06T00:08:54.920 INFO:teuthology.orchestra.run.smithi017.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-06T00:08:54.921 INFO:teuthology.orchestra.run.smithi017.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-06T00:08:54.922 INFO:teuthology.orchestra.run.smithi017.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-06T00:08:54.935 INFO:teuthology.orchestra.run.smithi066.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-06T00:08:54.936 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-06T00:08:54.937 INFO:teuthology.orchestra.run.smithi017.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-06T00:08:54.937 INFO:teuthology.orchestra.run.smithi066.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-06T00:08:54.938 INFO:teuthology.orchestra.run.smithi066.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-06T00:08:54.938 INFO:teuthology.orchestra.run.smithi066.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-06T00:08:54.938 INFO:teuthology.orchestra.run.smithi066.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-06T00:08:54.939 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-06T00:08:54.942 INFO:teuthology.orchestra.run.smithi017.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-06T00:08:54.957 INFO:teuthology.orchestra.run.smithi066.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-06T00:08:54.988 INFO:teuthology.orchestra.run.smithi017.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-06T00:08:54.989 INFO:teuthology.orchestra.run.smithi017.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-06T00:08:54.989 INFO:teuthology.orchestra.run.smithi017.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-06T00:08:54.990 INFO:teuthology.orchestra.run.smithi017.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-06T00:08:55.051 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-06T00:08:55.051 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB] 2022-03-06T00:08:55.054 INFO:teuthology.orchestra.run.smithi066.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-06T00:08:55.054 INFO:teuthology.orchestra.run.smithi066.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-06T00:08:55.055 INFO:teuthology.orchestra.run.smithi066.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-06T00:08:55.056 INFO:teuthology.orchestra.run.smithi066.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-06T00:08:55.056 INFO:teuthology.orchestra.run.smithi066.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-06T00:08:55.059 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [669 kB] 2022-03-06T00:08:55.066 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-06T00:08:55.067 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-06T00:08:55.068 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-06T00:08:55.068 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-06T00:08:55.069 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-06T00:08:55.069 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-06T00:08:55.091 INFO:teuthology.orchestra.run.smithi017.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-06T00:08:55.094 INFO:teuthology.orchestra.run.smithi017.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-06T00:08:55.094 INFO:teuthology.orchestra.run.smithi017.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [669 kB] 2022-03-06T00:08:55.101 INFO:teuthology.orchestra.run.smithi017.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB] 2022-03-06T00:08:55.107 INFO:teuthology.orchestra.run.smithi017.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-06T00:08:55.109 INFO:teuthology.orchestra.run.smithi017.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-06T00:08:55.114 INFO:teuthology.orchestra.run.smithi017.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,432 B] 2022-03-06T00:08:55.115 INFO:teuthology.orchestra.run.smithi017.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.7 kB] 2022-03-06T00:08:55.115 INFO:teuthology.orchestra.run.smithi017.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-06T00:08:55.157 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-06T00:08:55.158 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-06T00:08:55.160 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-06T00:08:55.160 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-06T00:08:55.207 INFO:teuthology.orchestra.run.smithi017.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-06T00:08:55.208 INFO:teuthology.orchestra.run.smithi017.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-06T00:08:55.209 INFO:teuthology.orchestra.run.smithi017.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-06T00:08:55.209 INFO:teuthology.orchestra.run.smithi017.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-06T00:08:55.210 INFO:teuthology.orchestra.run.smithi017.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-06T00:08:55.210 INFO:teuthology.orchestra.run.smithi017.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-06T00:08:55.211 INFO:teuthology.orchestra.run.smithi017.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-06T00:08:55.264 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-06T00:08:55.265 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-06T00:08:55.266 INFO:teuthology.orchestra.run.smithi007.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-06T00:08:55.316 INFO:teuthology.orchestra.run.smithi017.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-06T00:08:57.367 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 10.0 MB in 2s (5,812 kB/s) 2022-03-06T00:08:57.498 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 10.0 MB in 2s (5,377 kB/s) 2022-03-06T00:08:58.042 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 10.0 MB in 2s (4,478 kB/s) 2022-03-06T00:08:58.522 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-06T00:08:58.545 DEBUG:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:08:58.611 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-06T00:08:58.705 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-06T00:08:58.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:08:58.789 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-06T00:08:58.789 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-06T00:08:58.804 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-06T00:08:58.939 INFO:teuthology.orchestra.run.smithi066.stdout:The following additional packages will be installed: 2022-03-06T00:08:58.940 INFO:teuthology.orchestra.run.smithi066.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:08:58.940 INFO:teuthology.orchestra.run.smithi066.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-06T00:08:58.941 INFO:teuthology.orchestra.run.smithi066.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-06T00:08:58.942 INFO:teuthology.orchestra.run.smithi066.stdout:Suggested packages: 2022-03-06T00:08:58.942 INFO:teuthology.orchestra.run.smithi066.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-06T00:08:58.943 INFO:teuthology.orchestra.run.smithi066.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-06T00:08:58.980 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-06T00:08:58.980 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-06T00:08:59.103 INFO:teuthology.orchestra.run.smithi066.stdout:The following NEW packages will be installed: 2022-03-06T00:08:59.104 INFO:teuthology.orchestra.run.smithi066.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:08:59.104 INFO:teuthology.orchestra.run.smithi066.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-06T00:08:59.104 INFO:teuthology.orchestra.run.smithi066.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-06T00:08:59.105 INFO:teuthology.orchestra.run.smithi066.stdout: thermald upower usbmuxd 2022-03-06T00:08:59.132 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2022-03-06T00:08:59.132 INFO:teuthology.orchestra.run.smithi007.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:08:59.132 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-06T00:08:59.132 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-06T00:08:59.133 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2022-03-06T00:08:59.133 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-06T00:08:59.133 INFO:teuthology.orchestra.run.smithi007.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-06T00:08:59.175 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-06T00:08:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:08:59.175 INFO:teuthology.orchestra.run.smithi007.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-06T00:08:59.176 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-06T00:08:59.176 INFO:teuthology.orchestra.run.smithi007.stdout: thermald upower usbmuxd 2022-03-06T00:08:59.246 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2022-03-06T00:08:59.267 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:08:59.285 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 13 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:08:59.285 INFO:teuthology.orchestra.run.smithi066.stdout:Need to get 80.1 MB of archives. 2022-03-06T00:08:59.285 INFO:teuthology.orchestra.run.smithi066.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-06T00:08:59.285 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-06T00:08:59.340 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2022-03-06T00:08:59.358 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 13 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:08:59.359 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 80.1 MB of archives. 2022-03-06T00:08:59.359 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-06T00:08:59.359 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-06T00:08:59.515 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2022-03-06T00:08:59.516 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2022-03-06T00:08:59.572 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-06T00:08:59.595 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-06T00:08:59.609 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-06T00:08:59.622 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-06T00:08:59.643 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-06T00:08:59.644 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-06T00:08:59.665 INFO:teuthology.orchestra.run.smithi017.stdout:The following additional packages will be installed: 2022-03-06T00:08:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:08:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-06T00:08:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-06T00:08:59.667 INFO:teuthology.orchestra.run.smithi017.stdout:Suggested packages: 2022-03-06T00:08:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-06T00:08:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-06T00:08:59.687 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-06T00:08:59.710 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-06T00:08:59.720 INFO:teuthology.orchestra.run.smithi017.stdout:The following NEW packages will be installed: 2022-03-06T00:08:59.721 INFO:teuthology.orchestra.run.smithi017.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-06T00:08:59.721 INFO:teuthology.orchestra.run.smithi017.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-06T00:08:59.721 INFO:teuthology.orchestra.run.smithi017.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-06T00:08:59.721 INFO:teuthology.orchestra.run.smithi017.stdout: thermald upower usbmuxd 2022-03-06T00:08:59.724 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-06T00:08:59.743 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-06T00:08:59.904 INFO:teuthology.orchestra.run.smithi017.stdout:0 upgraded, 13 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:08:59.904 INFO:teuthology.orchestra.run.smithi017.stdout:Need to get 80.1 MB of archives. 2022-03-06T00:08:59.905 INFO:teuthology.orchestra.run.smithi017.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-06T00:08:59.905 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-06T00:09:00.178 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-06T00:09:00.208 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-06T00:09:00.234 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-06T00:09:00.288 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-06T00:09:00.331 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-06T00:09:00.568 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-06T00:09:00.594 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-06T00:09:00.890 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-06T00:09:00.949 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-06T00:09:01.368 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-06T00:09:01.720 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-06T00:09:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-06T00:09:03.102 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-06T00:09:03.103 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-06T00:09:03.107 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-06T00:09:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-06T00:09:03.220 INFO:teuthology.orchestra.run.smithi066.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-06T00:09:03.220 INFO:teuthology.orchestra.run.smithi066.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-06T00:09:03.221 INFO:teuthology.orchestra.run.smithi066.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-06T00:09:03.227 INFO:teuthology.orchestra.run.smithi066.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-06T00:09:03.285 INFO:teuthology.orchestra.run.smithi066.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-06T00:09:03.587 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 80.1 MB in 4s (20.1 MB/s) 2022-03-06T00:09:03.703 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 80.1 MB in 4s (19.2 MB/s) 2022-03-06T00:09:03.715 INFO:teuthology.orchestra.run.smithi017.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-06T00:09:03.715 INFO:teuthology.orchestra.run.smithi017.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-06T00:09:03.716 INFO:teuthology.orchestra.run.smithi017.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-06T00:09:03.722 INFO:teuthology.orchestra.run.smithi017.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-06T00:09:03.724 INFO:teuthology.orchestra.run.smithi017.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-06T00:09:03.801 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-06T00:09:03.897 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-06T00:09:04.132 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 80.1 MB in 4s (20.0 MB/s) 2022-03-06T00:09:04.333 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-06T00:09:05.294 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-06T00:09:05.299 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-06T00:09:05.379 INFO:teuthology.orchestra.run.smithi066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-06T00:09:05.386 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-06T00:09:05.448 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:09:05.535 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:09:05.818 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-06T00:09:05.821 INFO:teuthology.orchestra.run.smithi017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-06T00:09:05.826 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-06T00:09:05.834 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-06T00:09:05.877 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:09:05.913 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-06T00:09:05.927 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-06T00:09:05.966 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:09:05.971 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:09:06.316 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-06T00:09:06.319 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-06T00:09:06.330 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-06T00:09:06.332 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-06T00:09:06.369 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:09:06.390 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:09:06.396 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-06T00:09:06.410 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-06T00:09:06.446 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:09:06.655 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-06T00:09:06.661 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-06T00:09:06.668 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-06T00:09:06.675 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-06T00:09:06.698 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:09:06.710 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:09:06.916 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-06T00:09:06.930 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-06T00:09:06.974 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:09:07.050 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-06T00:09:07.063 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-06T00:09:07.071 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-06T00:09:07.086 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-06T00:09:07.099 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:09:07.130 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:09:07.286 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-06T00:09:07.300 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-06T00:09:07.336 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:09:07.441 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-06T00:09:07.452 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-06T00:09:07.454 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:07.466 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-06T00:09:07.492 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:07.511 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:09:07.689 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-06T00:09:07.702 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:07.739 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:07.939 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-06T00:09:07.953 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:07.988 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:09.858 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-06T00:09:09.875 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:09.967 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:10.048 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-06T00:09:10.062 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:10.171 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:10.180 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-06T00:09:10.196 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:10.302 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:10.462 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-06T00:09:10.475 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:10.520 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:10.670 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-06T00:09:10.682 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:10.743 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:10.799 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-06T00:09:10.812 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-06T00:09:10.849 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:16.303 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-06T00:09:16.322 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-06T00:09:16.384 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:09:16.605 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-06T00:09:16.621 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-06T00:09:16.663 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:09:16.709 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-06T00:09:16.722 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-06T00:09:16.741 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-06T00:09:16.756 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-06T00:09:16.767 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:09:16.791 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:09:16.982 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-06T00:09:16.999 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-06T00:09:17.041 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:09:17.078 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-06T00:09:17.093 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-06T00:09:17.136 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:09:17.137 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package thermald. 2022-03-06T00:09:17.151 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-06T00:09:17.195 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:09:17.411 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package thermald. 2022-03-06T00:09:17.425 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-06T00:09:17.469 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:09:17.497 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package thermald. 2022-03-06T00:09:17.511 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-06T00:09:17.556 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:09:17.744 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package upower. 2022-03-06T00:09:17.761 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-06T00:09:17.817 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-06T00:09:17.977 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package upower. 2022-03-06T00:09:17.991 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-06T00:09:18.045 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-06T00:09:18.096 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package upower. 2022-03-06T00:09:18.113 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-06T00:09:18.165 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-06T00:09:18.261 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package usbmuxd. 2022-03-06T00:09:18.275 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-06T00:09:18.319 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:09:18.444 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package usbmuxd. 2022-03-06T00:09:18.460 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-06T00:09:18.503 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:09:18.580 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package usbmuxd. 2022-03-06T00:09:18.596 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-06T00:09:18.611 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:09:18.630 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:09:18.765 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:09:18.827 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:09:18.891 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:09:18.903 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-06T00:09:18.949 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:09:19.017 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:09:19.026 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-06T00:09:19.075 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:09:19.143 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:09:19.143 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-06T00:09:19.203 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:09:19.252 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-06T00:09:19.287 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:09:19.344 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:09:19.378 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-06T00:09:19.404 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-06T00:09:19.470 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:09:19.538 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-06T00:09:19.597 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-06T00:09:19.641 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-06T00:09:19.859 INFO:teuthology.orchestra.run.smithi066.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-06T00:09:19.905 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:09:20.016 INFO:teuthology.orchestra.run.smithi007.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-06T00:09:20.073 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:09:20.078 INFO:teuthology.orchestra.run.smithi017.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-06T00:09:20.093 INFO:teuthology.orchestra.run.smithi066.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-06T00:09:20.101 INFO:teuthology.orchestra.run.smithi066.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-06T00:09:20.102 INFO:teuthology.orchestra.run.smithi066.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-06T00:09:20.132 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-06T00:09:20.239 INFO:teuthology.orchestra.run.smithi007.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-06T00:09:20.252 INFO:teuthology.orchestra.run.smithi007.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-06T00:09:20.253 INFO:teuthology.orchestra.run.smithi007.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-06T00:09:20.308 INFO:teuthology.orchestra.run.smithi017.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-06T00:09:20.319 INFO:teuthology.orchestra.run.smithi017.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-06T00:09:20.319 INFO:teuthology.orchestra.run.smithi017.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-06T00:09:20.738 INFO:teuthology.orchestra.run.smithi066.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-06T00:09:20.739 INFO:teuthology.orchestra.run.smithi007.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-06T00:09:20.901 INFO:teuthology.orchestra.run.smithi017.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-06T00:09:20.974 INFO:teuthology.orchestra.run.smithi066.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-06T00:09:20.980 INFO:teuthology.orchestra.run.smithi007.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-06T00:09:21.018 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:09:21.044 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:09:21.140 INFO:teuthology.orchestra.run.smithi017.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-06T00:09:21.203 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-06T00:09:21.333 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:09:21.334 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:09:21.346 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:09:21.347 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:09:21.499 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:09:21.499 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-06T00:09:21.824 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:21.831 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:22.008 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:23.107 INFO:teuthology.orchestra.run.smithi066.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-06T00:09:23.108 INFO:teuthology.orchestra.run.smithi066.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-06T00:09:23.108 INFO:teuthology.orchestra.run.smithi007.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-06T00:09:23.109 INFO:teuthology.orchestra.run.smithi007.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-06T00:09:23.277 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:23.283 INFO:teuthology.orchestra.run.smithi017.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-06T00:09:23.284 INFO:teuthology.orchestra.run.smithi017.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-06T00:09:23.291 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:23.476 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:24.636 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:24.667 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:24.861 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:26.021 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:09:26.147 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-06T00:09:26.286 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:09:26.403 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-06T00:09:26.868 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-06T00:09:27.147 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-06T00:09:30.367 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:09:30.595 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:09:30.777 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:09:31.035 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-06T00:09:31.263 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-06T00:09:31.364 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:31.495 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-06T00:09:31.553 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-06T00:09:31.554 INFO:teuthology.orchestra.run.smithi007.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:09:31.604 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:31.661 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-06T00:09:31.788 INFO:teuthology.orchestra.run.smithi017.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-06T00:09:31.788 INFO:teuthology.orchestra.run.smithi017.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:09:31.860 INFO:teuthology.orchestra.run.smithi066.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-06T00:09:31.861 INFO:teuthology.orchestra.run.smithi066.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:09:55.507 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-06T00:09:55.508 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub' 2022-03-06T00:09:55.516 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:09:55.543 INFO:teuthology.orchestra.run.smithi007.stdout:Generating grub configuration file ... 2022-03-06T00:09:55.636 INFO:teuthology.orchestra.run.smithi017.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-06T00:09:55.637 INFO:teuthology.orchestra.run.smithi017.stdout:Sourcing file `/etc/default/grub' 2022-03-06T00:09:55.642 INFO:teuthology.orchestra.run.smithi017.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:09:55.665 INFO:teuthology.orchestra.run.smithi017.stdout:Generating grub configuration file ... 2022-03-06T00:09:56.871 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:09:56.871 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:09:56.872 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:09:56.872 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:09:56.872 INFO:teuthology.orchestra.run.smithi017.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:09:56.873 INFO:teuthology.orchestra.run.smithi017.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:09:56.873 INFO:teuthology.orchestra.run.smithi017.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:09:56.873 INFO:teuthology.orchestra.run.smithi017.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:09:56.874 INFO:teuthology.orchestra.run.smithi066.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-06T00:09:56.875 INFO:teuthology.orchestra.run.smithi066.stdout:Sourcing file `/etc/default/grub' 2022-03-06T00:09:56.876 INFO:teuthology.orchestra.run.smithi066.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:09:56.876 INFO:teuthology.orchestra.run.smithi066.stdout:Generating grub configuration file ... 2022-03-06T00:09:56.876 INFO:teuthology.orchestra.run.smithi066.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:09:56.877 INFO:teuthology.orchestra.run.smithi066.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:09:56.877 INFO:teuthology.orchestra.run.smithi066.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:09:56.877 INFO:teuthology.orchestra.run.smithi066.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:09:58.436 INFO:teuthology.orchestra.run.smithi066.stdout:done 2022-03-06T00:09:58.780 DEBUG:teuthology.orchestra.run.smithi066:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:09:58.806 INFO:teuthology.orchestra.run.smithi066.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:09:58.806 INFO:teuthology.orchestra.run.smithi066.stdout:Status: install ok installed 2022-03-06T00:09:58.806 INFO:teuthology.orchestra.run.smithi066.stdout:Priority: optional 2022-03-06T00:09:58.807 INFO:teuthology.orchestra.run.smithi066.stdout:Section: kernel 2022-03-06T00:09:58.807 INFO:teuthology.orchestra.run.smithi066.stdout:Installed-Size: 19 2022-03-06T00:09:58.807 INFO:teuthology.orchestra.run.smithi066.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:09:58.807 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: amd64 2022-03-06T00:09:58.808 INFO:teuthology.orchestra.run.smithi066.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:09:58.808 INFO:teuthology.orchestra.run.smithi066.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:09:58.808 INFO:teuthology.orchestra.run.smithi066.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:09:58.808 INFO:teuthology.orchestra.run.smithi066.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:09:58.809 INFO:teuthology.orchestra.run.smithi066.stdout:Recommends: thermald 2022-03-06T00:09:58.809 INFO:teuthology.orchestra.run.smithi066.stdout:Description: Generic Linux kernel image 2022-03-06T00:09:58.809 INFO:teuthology.orchestra.run.smithi066.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:09:58.809 INFO:teuthology.orchestra.run.smithi066.stdout: available. 2022-03-06T00:09:58.810 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-06T00:09:58.810 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-06T00:09:58.810 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-06T00:09:58.811 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-06T00:09:58.811 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2022-03-06T00:09:58.811 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt-get clean 2022-03-06T00:09:58.909 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt-get update 2022-03-06T00:09:58.966 INFO:teuthology.orchestra.run.smithi017.stdout:done 2022-03-06T00:09:58.985 INFO:teuthology.orchestra.run.smithi007.stdout:done 2022-03-06T00:09:59.254 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:09:59.255 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:09:59.325 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:09:59.364 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:09:59.395 DEBUG:teuthology.orchestra.run.smithi017:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:09:59.396 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:09:59.396 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2022-03-06T00:09:59.397 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2022-03-06T00:09:59.397 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2022-03-06T00:09:59.397 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 19 2022-03-06T00:09:59.397 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:09:59.398 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2022-03-06T00:09:59.398 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:09:59.398 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:09:59.398 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:09:59.399 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:09:59.399 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2022-03-06T00:09:59.399 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2022-03-06T00:09:59.399 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:09:59.399 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2022-03-06T00:09:59.400 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-06T00:09:59.400 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-06T00:09:59.400 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-06T00:09:59.401 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-06T00:09:59.402 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2022-03-06T00:09:59.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2022-03-06T00:09:59.407 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:09:59.428 INFO:teuthology.orchestra.run.smithi017.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:09:59.428 INFO:teuthology.orchestra.run.smithi017.stdout:Status: install ok installed 2022-03-06T00:09:59.428 INFO:teuthology.orchestra.run.smithi017.stdout:Priority: optional 2022-03-06T00:09:59.428 INFO:teuthology.orchestra.run.smithi017.stdout:Section: kernel 2022-03-06T00:09:59.429 INFO:teuthology.orchestra.run.smithi017.stdout:Installed-Size: 19 2022-03-06T00:09:59.429 INFO:teuthology.orchestra.run.smithi017.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:09:59.429 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: amd64 2022-03-06T00:09:59.429 INFO:teuthology.orchestra.run.smithi017.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:09:59.430 INFO:teuthology.orchestra.run.smithi017.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:09:59.430 INFO:teuthology.orchestra.run.smithi017.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:09:59.430 INFO:teuthology.orchestra.run.smithi017.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:09:59.430 INFO:teuthology.orchestra.run.smithi017.stdout:Recommends: thermald 2022-03-06T00:09:59.430 INFO:teuthology.orchestra.run.smithi017.stdout:Description: Generic Linux kernel image 2022-03-06T00:09:59.431 INFO:teuthology.orchestra.run.smithi017.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:09:59.431 INFO:teuthology.orchestra.run.smithi017.stdout: available. 2022-03-06T00:09:59.431 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-06T00:09:59.432 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-06T00:09:59.432 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-06T00:09:59.432 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-06T00:09:59.432 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2022-03-06T00:09:59.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get clean 2022-03-06T00:09:59.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2022-03-06T00:09:59.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt-get update 2022-03-06T00:09:59.682 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:09:59.740 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:09:59.741 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:09:59.797 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:09:59.815 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:09:59.880 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:09:59.902 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:09:59.962 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:10:00.841 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-06T00:10:00.865 DEBUG:teuthology.orchestra.run.smithi066:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:10:00.936 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-06T00:10:01.113 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-06T00:10:01.114 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-06T00:10:01.229 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2022-03-06T00:10:01.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:10:01.303 INFO:teuthology.orchestra.run.smithi066.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-06T00:10:01.303 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 0 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:10:01.305 DEBUG:teuthology.orchestra.run.smithi066:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:10:01.310 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-06T00:10:01.320 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2022-03-06T00:10:01.324 INFO:teuthology.orchestra.run.smithi066.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:10:01.324 INFO:teuthology.orchestra.run.smithi066.stdout:Status: install ok installed 2022-03-06T00:10:01.325 INFO:teuthology.orchestra.run.smithi066.stdout:Priority: optional 2022-03-06T00:10:01.325 INFO:teuthology.orchestra.run.smithi066.stdout:Section: kernel 2022-03-06T00:10:01.325 INFO:teuthology.orchestra.run.smithi066.stdout:Installed-Size: 19 2022-03-06T00:10:01.325 INFO:teuthology.orchestra.run.smithi066.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:10:01.325 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: amd64 2022-03-06T00:10:01.326 INFO:teuthology.orchestra.run.smithi066.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:10:01.326 INFO:teuthology.orchestra.run.smithi066.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:10:01.326 INFO:teuthology.orchestra.run.smithi066.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:10:01.326 INFO:teuthology.orchestra.run.smithi066.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:10:01.327 INFO:teuthology.orchestra.run.smithi066.stdout:Recommends: thermald 2022-03-06T00:10:01.327 INFO:teuthology.orchestra.run.smithi066.stdout:Description: Generic Linux kernel image 2022-03-06T00:10:01.327 INFO:teuthology.orchestra.run.smithi066.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:10:01.327 INFO:teuthology.orchestra.run.smithi066.stdout: available. 2022-03-06T00:10:01.328 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2022-03-06T00:10:01.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-06T00:10:01.336 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.IZfVUpSA2B 2022-03-06T00:10:01.337 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub/grub.cfg /tmp/tmp.IZfVUpSA2B 2022-03-06T00:10:01.349 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.IZfVUpSA2B 2022-03-06T00:10:01.402 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-06T00:10:01.446 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.IZfVUpSA2B is 8KB 2022-03-06T00:10:01.461 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.IZfVUpSA2B 2022-03-06T00:10:01.468 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-06T00:10:01.477 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2022-03-06T00:10:01.478 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2022-03-06T00:10:01.523 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:10:01.524 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-06T00:10:01.524 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-06T00:10:01.558 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-06T00:10:01.559 DEBUG:teuthology.orchestra.run.smithi066:> sudo update-grub 2022-03-06T00:10:01.578 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-06T00:10:01.579 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-06T00:10:01.652 INFO:teuthology.orchestra.run.smithi017.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-06T00:10:01.653 INFO:teuthology.orchestra.run.smithi017.stdout:0 upgraded, 0 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:10:01.654 DEBUG:teuthology.orchestra.run.smithi017:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:10:01.671 INFO:teuthology.orchestra.run.smithi017.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:10:01.671 INFO:teuthology.orchestra.run.smithi017.stdout:Status: install ok installed 2022-03-06T00:10:01.672 INFO:teuthology.orchestra.run.smithi017.stdout:Priority: optional 2022-03-06T00:10:01.672 INFO:teuthology.orchestra.run.smithi017.stdout:Section: kernel 2022-03-06T00:10:01.672 INFO:teuthology.orchestra.run.smithi017.stdout:Installed-Size: 19 2022-03-06T00:10:01.672 INFO:teuthology.orchestra.run.smithi017.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:10:01.673 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: amd64 2022-03-06T00:10:01.673 INFO:teuthology.orchestra.run.smithi017.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:10:01.673 INFO:teuthology.orchestra.run.smithi017.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:10:01.673 INFO:teuthology.orchestra.run.smithi017.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:10:01.673 INFO:teuthology.orchestra.run.smithi017.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:10:01.674 INFO:teuthology.orchestra.run.smithi017.stdout:Recommends: thermald 2022-03-06T00:10:01.674 INFO:teuthology.orchestra.run.smithi017.stdout:Description: Generic Linux kernel image 2022-03-06T00:10:01.674 INFO:teuthology.orchestra.run.smithi017.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:10:01.674 INFO:teuthology.orchestra.run.smithi017.stdout: available. 2022-03-06T00:10:01.675 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2022-03-06T00:10:01.681 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.baFiZgE8vg 2022-03-06T00:10:01.682 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub/grub.cfg /tmp/tmp.baFiZgE8vg 2022-03-06T00:10:01.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.baFiZgE8vg 2022-03-06T00:10:01.765 INFO:teuthology.orchestra.run.smithi007.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-06T00:10:01.765 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 0 newly installed, 0 to remove and 73 not upgraded. 2022-03-06T00:10:01.767 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic-hwe-20.04 2022-03-06T00:10:01.788 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic-hwe-20.04 2022-03-06T00:10:01.788 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2022-03-06T00:10:01.789 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2022-03-06T00:10:01.789 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2022-03-06T00:10:01.789 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 19 2022-03-06T00:10:01.789 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2022-03-06T00:10:01.790 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2022-03-06T00:10:01.790 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta-hwe-5.13 2022-03-06T00:10:01.790 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-06T00:10:01.791 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-06T00:10:01.791 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-06T00:10:01.791 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2022-03-06T00:10:01.792 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2022-03-06T00:10:01.792 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2022-03-06T00:10:01.792 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2022-03-06T00:10:01.792 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2022-03-06T00:10:01.820 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.baFiZgE8vg is 8KB 2022-03-06T00:10:01.837 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.baFiZgE8vg 2022-03-06T00:10:01.838 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.iIoKPOFBYs 2022-03-06T00:10:01.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub/grub.cfg /tmp/tmp.iIoKPOFBYs 2022-03-06T00:10:01.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-06T00:10:01.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.iIoKPOFBYs 2022-03-06T00:10:01.904 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:10:01.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-06T00:10:01.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-06T00:10:01.948 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.iIoKPOFBYs is 8KB 2022-03-06T00:10:01.954 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-06T00:10:01.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo update-grub 2022-03-06T00:10:01.963 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.iIoKPOFBYs 2022-03-06T00:10:01.969 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-06T00:10:02.032 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:10:02.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-06T00:10:02.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-06T00:10:02.070 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-06T00:10:02.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo update-grub 2022-03-06T00:10:02.739 INFO:teuthology.orchestra.run.smithi066.stderr:Sourcing file `/etc/default/grub' 2022-03-06T00:10:02.740 INFO:teuthology.orchestra.run.smithi066.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:10:02.752 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2022-03-06T00:10:02.973 INFO:teuthology.orchestra.run.smithi066.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:10:02.984 INFO:teuthology.orchestra.run.smithi066.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:10:03.141 INFO:teuthology.orchestra.run.smithi017.stderr:Sourcing file `/etc/default/grub' 2022-03-06T00:10:03.151 INFO:teuthology.orchestra.run.smithi017.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:10:03.191 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2022-03-06T00:10:03.287 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2022-03-06T00:10:03.288 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-06T00:10:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2022-03-06T00:10:03.376 INFO:teuthology.orchestra.run.smithi066.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:10:03.387 INFO:teuthology.orchestra.run.smithi066.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:10:03.449 INFO:teuthology.orchestra.run.smithi017.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:10:03.462 INFO:teuthology.orchestra.run.smithi017.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:10:03.552 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-06T00:10:03.564 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-06T00:10:03.797 INFO:teuthology.orchestra.run.smithi017.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:10:03.809 INFO:teuthology.orchestra.run.smithi017.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:10:03.924 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-06T00:10:03.929 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-06T00:10:04.117 INFO:teuthology.orchestra.run.smithi066.stderr:done 2022-03-06T00:10:04.124 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2022-03-06T00:10:05.370 INFO:teuthology.orchestra.run.smithi017.stderr:done 2022-03-06T00:10:05.372 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2022-03-06T00:10:05.505 INFO:teuthology.orchestra.run.smithi007.stderr:done 2022-03-06T00:10:05.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2022-03-06T00:10:34.163 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-06T00:10:34.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-06T00:10:34.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:10:35.377 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-06T00:10:35.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2022-03-06T00:10:35.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:10:35.510 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-06T00:10:35.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-06T00:10:35.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:10:52.495 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2022-03-06T00:10:53.771 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2022-03-06T00:10:54.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2022-03-06T00:11:22.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-06T00:11:22.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:11:23.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2022-03-06T00:11:23.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:11:24.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-06T00:11:24.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:11:25.579 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2022-03-06T00:11:26.826 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2022-03-06T00:11:39.848 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-06T00:11:40.362 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-06T00:11:40.363 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-03-06T00:11:40.363 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2022-03-06T00:11:40.411 INFO:teuthology.orchestra.run.smithi007.stdout:5.13.0-30-generic 2022-03-06T00:11:40.412 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-06T00:11:40.412 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-06T00:11:40.412 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-06T00:11:41.414 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-06T00:11:41.415 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-06T00:11:41.463 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2022-03-06T00:11:41.487 DEBUG:teuthology.parallel:result is None 2022-03-06T00:11:55.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-06T00:11:55.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:11:56.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2022-03-06T00:11:56.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:11:57.079 DEBUG:teuthology.orchestra.run.smithi017:> true 2022-03-06T00:11:57.596 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2022-03-06T00:11:57.597 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-30-generic"... 2022-03-06T00:11:57.597 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2022-03-06T00:11:57.602 INFO:teuthology.orchestra.run.smithi017.stdout:5.13.0-30-generic 2022-03-06T00:11:57.602 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-06T00:11:57.603 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-06T00:11:57.603 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-06T00:11:58.605 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-06T00:11:58.605 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-06T00:11:58.683 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2022-03-06T00:11:58.707 DEBUG:teuthology.parallel:result is None 2022-03-06T00:12:26.959 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2022-03-06T00:12:56.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-06T00:12:56.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:12:57.155 DEBUG:teuthology.orchestra.run.smithi066:> true 2022-03-06T00:12:57.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-06T00:12:57.618 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-30-generic"... 2022-03-06T00:12:57.618 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2022-03-06T00:12:57.667 INFO:teuthology.orchestra.run.smithi066.stdout:5.13.0-30-generic 2022-03-06T00:12:57.668 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-06T00:12:57.668 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-06T00:12:57.668 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-06T00:12:58.671 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-06T00:12:58.672 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-06T00:12:58.741 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2022-03-06T00:12:58.759 DEBUG:teuthology.parallel:result is None 2022-03-06T00:12:58.760 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-06T00:12:58.804 INFO:teuthology.task.internal:Creating test directory... 2022-03-06T00:12:58.805 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-06T00:12:58.808 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-06T00:12:58.811 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-06T00:12:58.818 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-06T00:12:58.839 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-06T00:12:58.851 INFO:teuthology.task.internal:Creating archive directory... 2022-03-06T00:12:58.852 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-06T00:12:58.855 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-06T00:12:58.857 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-06T00:12:58.891 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-06T00:12:58.904 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-06T00:12:58.905 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-06T00:12:58.913 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-06T00:12:58.921 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-06T00:12:58.935 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:12:58.942 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:12:58.951 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:12:58.961 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:12:58.961 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:12:58.971 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-06T00:12:58.973 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-06T00:12:58.996 INFO:teuthology.task.internal:Configuring sudo... 2022-03-06T00:12:58.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-06T00:12:59.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-06T00:12:59.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-06T00:12:59.019 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-06T00:12:59.036 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-06T00:12:59.037 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-06T00:12:59.041 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-06T00:12:59.044 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-06T00:12:59.068 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-06T00:12:59.095 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-06T00:12:59.101 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:12:59.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-06T00:12:59.121 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-06T00:12:59.129 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-06T00:12:59.137 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:12:59.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-06T00:12:59.211 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-06T00:12:59.220 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-06T00:12:59.268 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:12:59.268 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-06T00:12:59.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2022-03-06T00:12:59.295 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2022-03-06T00:12:59.298 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2022-03-06T00:12:59.344 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-06T00:12:59.447 INFO:teuthology.task.internal:Starting timer... 2022-03-06T00:12:59.448 INFO:teuthology.run_tasks:Running task pcp... 2022-03-06T00:12:59.463 INFO:teuthology.run_tasks:Running task selinux... 2022-03-06T00:12:59.477 DEBUG:teuthology.task.selinux:Excluding smithi007: OS 'ubuntu' does not support SELinux 2022-03-06T00:12:59.478 DEBUG:teuthology.task.selinux:Excluding smithi017: OS 'ubuntu' does not support SELinux 2022-03-06T00:12:59.478 DEBUG:teuthology.task.selinux:Excluding smithi066: OS 'ubuntu' does not support SELinux 2022-03-06T00:12:59.478 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-06T00:12:59.479 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-06T00:12:59.479 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-06T00:12:59.479 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-06T00:12:59.548 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-03-06T00:12:59.554 INFO:teuthology.repo_utils:Fetching master from origin 2022-03-06T00:12:59.589 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-06T00:12:59.609 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-06T00:12:59.611 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi017.front.sepia.ceph.com,smithi066.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-06T00:16:08.902 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi066.front.sepia.ceph.com')] 2022-03-06T00:16:08.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2022-03-06T00:16:08.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:16:08.959 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-06T00:16:09.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2022-03-06T00:16:09.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:16:09.134 DEBUG:teuthology.orchestra.run.smithi017:> true 2022-03-06T00:16:09.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-03-06T00:16:09.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-06T00:16:09.303 DEBUG:teuthology.orchestra.run.smithi066:> true 2022-03-06T00:16:09.429 INFO:teuthology.run_tasks:Running task clock... 2022-03-06T00:16:09.445 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-06T00:16:09.445 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-06T00:16:09.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-06T00:16:09.450 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-06T00:16:09.450 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-06T00:16:09.453 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-06T00:16:09.453 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-06T00:16:09.483 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-06T00:16:09.483 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: Command line: ntpd -gq 2022-03-06T00:16:09.485 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-06T00:16:09.485 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: Command line: ntpd -gq 2022-03-06T00:16:09.485 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: proto: precision = 0.060 usec (-24) 2022-03-06T00:16:09.486 INFO:teuthology.orchestra.run.smithi007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:16:09.487 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:16:09.487 INFO:teuthology.orchestra.run.smithi007.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:16:09.488 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:16:09.489 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: Listen and drop on 0 v6wildcard [::]:123 2022-03-06T00:16:09.489 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-06T00:16:09.489 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: Listen normally on 2 lo 127.0.0.1:123 2022-03-06T00:16:09.489 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: Listen normally on 3 ens1f0 172.21.15.7:123 2022-03-06T00:16:09.490 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: Listen normally on 4 lo [::1]:123 2022-03-06T00:16:09.490 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15ea%4]:123 2022-03-06T00:16:09.490 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:09 ntpd[8946]: Listening on routing socket on fd #22 for interface updates 2022-03-06T00:16:09.491 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: proto: precision = 0.076 usec (-24) 2022-03-06T00:16:09.492 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-06T00:16:09.492 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: Command line: ntpd -gq 2022-03-06T00:16:09.492 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: proto: precision = 0.063 usec (-24) 2022-03-06T00:16:09.493 INFO:teuthology.orchestra.run.smithi017.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:16:09.493 INFO:teuthology.orchestra.run.smithi066.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:16:09.494 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:16:09.494 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-06T00:16:09.495 INFO:teuthology.orchestra.run.smithi017.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:16:09.496 INFO:teuthology.orchestra.run.smithi066.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:16:09.498 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:16:09.498 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: Listen and drop on 0 v6wildcard [::]:123 2022-03-06T00:16:09.498 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-06T00:16:09.498 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: Listen normally on 2 lo 127.0.0.1:123 2022-03-06T00:16:09.499 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: Listen normally on 3 ens1f0 172.21.15.17:123 2022-03-06T00:16:09.499 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: Listen normally on 4 lo [::1]:123 2022-03-06T00:16:09.499 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe59:d992%4]:123 2022-03-06T00:16:09.499 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:09 ntpd[8927]: Listening on routing socket on fd #22 for interface updates 2022-03-06T00:16:09.500 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: restrict ::: KOD does nothing without LIMITED. 2022-03-06T00:16:09.500 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: Listen and drop on 0 v6wildcard [::]:123 2022-03-06T00:16:09.500 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-06T00:16:09.500 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: Listen normally on 2 lo 127.0.0.1:123 2022-03-06T00:16:09.500 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: Listen normally on 3 enp3s0f1 172.21.15.66:123 2022-03-06T00:16:09.501 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: Listen normally on 4 lo [::1]:123 2022-03-06T00:16:09.501 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8bd%5]:123 2022-03-06T00:16:09.501 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:09 ntpd[9035]: Listening on routing socket on fd #22 for interface updates 2022-03-06T00:16:10.484 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:10 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:10.484 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:10 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:10.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:10 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:10.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:10 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:10.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:10 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:10.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:10 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:10.489 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:10.489 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:10.489 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:10.489 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:10.490 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:10.490 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:10.490 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:10.490 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:10.491 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:10.491 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:10.491 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:10.491 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:10 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:11.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.486 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.486 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.486 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.487 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.487 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:11.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:11.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:11.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:11 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:11.492 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:11 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.492 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:11 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.493 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:11 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.493 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:11 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.493 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:11 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:11.493 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:11 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:11.495 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.495 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.495 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.495 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.496 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.496 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.496 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:11.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:11.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:11.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:11.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:11.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:11 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:12.484 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:12 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:12.484 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:12 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:12.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:12 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:12.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:12 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:12.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:12 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:12.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:12 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:12.486 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:12 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:12.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:12 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:12.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:12 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:12.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:12 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:12.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:12 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:12.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:12 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:12.489 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:12.489 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:12.490 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:12.490 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:12.490 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:12.490 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:12.491 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:12.491 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:12.491 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:12.491 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:12.492 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:12.492 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:12 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:13.487 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.487 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:13.490 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:13.490 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.491 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.491 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:13.492 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:13 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:13.501 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.501 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.501 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.501 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.502 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.502 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.503 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:13.503 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:13.504 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.504 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.504 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:13.505 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:13 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:13.508 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.508 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.508 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.509 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.509 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.509 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.509 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.509 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.510 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:13.510 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:13.510 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.511 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.511 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:13.512 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:13.512 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:13.512 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:13.512 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:13.512 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:13 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:14.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:14 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:14 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:14 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:14 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:14 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:14.489 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:14 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:14.490 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.490 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.490 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.491 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.491 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.491 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.491 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:14.492 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:14.493 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.493 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.493 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:14.494 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:14 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:14.496 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.496 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.497 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.498 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.498 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.498 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.499 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:14.499 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:14.499 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:14.500 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:14.500 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:14.500 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:14 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:15.485 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.486 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.486 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.487 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.487 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:15.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:15.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:15.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:15 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:15.492 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.492 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.493 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.493 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.493 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.493 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.494 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:15.494 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:15.494 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.494 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.494 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:15.495 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:15 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:15.496 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.496 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.496 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.497 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.498 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:15.498 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:15.499 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.499 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.499 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:15.499 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:15.499 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:15.500 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:15.500 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:15.500 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:15 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:16.486 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.487 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.487 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.488 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:16.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:16.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:16.489 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:16.490 INFO:teuthology.orchestra.run.smithi007.stdout: 6 Mar 00:16:16 ntpd[8946]: ntpd: time slew -0.001140 s 2022-03-06T00:16:16.490 INFO:teuthology.orchestra.run.smithi007.stdout:ntpd: time slew -0.001140s 2022-03-06T00:16:16.491 INFO:teuthology.orchestra.run.smithi007.stderr: 6 Mar 00:16:16 ntpd[8946]: can't open /var/log/ntpstats/loopstats.20220306: Permission denied 2022-03-06T00:16:16.493 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:16 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.493 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:16 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.493 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:16 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.493 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:16 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.494 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:16 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:16.494 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:16 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:16.494 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.495 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.495 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.495 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.495 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:16.495 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:16.496 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:16.496 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:16.496 INFO:teuthology.orchestra.run.smithi066.stdout: 6 Mar 00:16:16 ntpd[9035]: ntpd: time slew +0.000691 s 2022-03-06T00:16:16.497 INFO:teuthology.orchestra.run.smithi066.stdout:ntpd: time slew +0.000691s 2022-03-06T00:16:16.497 INFO:teuthology.orchestra.run.smithi066.stderr: 6 Mar 00:16:16 ntpd[9035]: can't open /var/log/ntpstats/loopstats.20220306: Permission denied 2022-03-06T00:16:16.540 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2022-03-06T00:16:16.541 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2022-03-06T00:16:16.541 INFO:teuthology.orchestra.run.smithi007.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:16.542 INFO:teuthology.orchestra.run.smithi007.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:16.542 INFO:teuthology.orchestra.run.smithi007.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:16.542 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:16.545 INFO:teuthology.orchestra.run.smithi066.stdout: remote refid st t when poll reach delay offset jitter 2022-03-06T00:16:16.546 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================== 2022-03-06T00:16:16.546 INFO:teuthology.orchestra.run.smithi066.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:16.546 INFO:teuthology.orchestra.run.smithi066.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:16.546 INFO:teuthology.orchestra.run.smithi066.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:16.547 INFO:teuthology.orchestra.run.smithi066.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:17.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:17.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:17.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:17.487 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:17.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:17.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:17.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:17.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:17.488 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:17.489 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:17.489 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-06T00:16:17.489 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: can't open /var/log/ntpstats/rawstats.20220306: Permission denied 2022-03-06T00:16:17.489 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-06T00:16:17.490 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: can't open /var/log/ntpstats/peerstats.20220306: Permission denied 2022-03-06T00:16:17.490 INFO:teuthology.orchestra.run.smithi017.stdout: 6 Mar 00:16:17 ntpd[8927]: ntpd: time slew -0.000726 s 2022-03-06T00:16:17.490 INFO:teuthology.orchestra.run.smithi017.stdout:ntpd: time slew -0.000726s 2022-03-06T00:16:17.491 INFO:teuthology.orchestra.run.smithi017.stderr: 6 Mar 00:16:17 ntpd[8927]: can't open /var/log/ntpstats/loopstats.20220306: Permission denied 2022-03-06T00:16:17.542 INFO:teuthology.orchestra.run.smithi017.stdout: remote refid st t when poll reach delay offset jitter 2022-03-06T00:16:17.542 INFO:teuthology.orchestra.run.smithi017.stdout:============================================================================== 2022-03-06T00:16:17.542 INFO:teuthology.orchestra.run.smithi017.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:17.542 INFO:teuthology.orchestra.run.smithi017.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:17.543 INFO:teuthology.orchestra.run.smithi017.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:17.543 INFO:teuthology.orchestra.run.smithi017.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-06T00:16:17.544 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-06T00:16:17.560 INFO:tasks.kubeadm:Kubeadm start 2022-03-06T00:16:17.561 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-03-06T00:16:17.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-06T00:16:17.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-06T00:16:17.568 DEBUG:teuthology.orchestra.run.smithi066:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-06T00:16:17.588 INFO:teuthology.orchestra.run.smithi017.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-06T00:16:17.590 INFO:teuthology.orchestra.run.smithi007.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-06T00:16:17.592 INFO:teuthology.orchestra.run.smithi066.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-06T00:16:17.598 INFO:teuthology.orchestra.run.smithi017.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-06T00:16:17.600 INFO:teuthology.orchestra.run.smithi007.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-06T00:16:17.603 INFO:teuthology.orchestra.run.smithi066.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-06T00:16:17.607 INFO:teuthology.orchestra.run.smithi017.stdout:net.ipv4.ip_forward = 1 2022-03-06T00:16:17.610 INFO:teuthology.orchestra.run.smithi007.stdout:net.ipv4.ip_forward = 1 2022-03-06T00:16:17.610 INFO:teuthology.orchestra.run.smithi066.stdout:net.ipv4.ip_forward = 1 2022-03-06T00:16:17.664 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:16:17.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-06T00:16:17.679 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:16:17.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/docker/daemon.json 2022-03-06T00:16:17.736 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:16:17.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-06T00:16:17.752 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:16:17.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/docker/daemon.json 2022-03-06T00:16:17.764 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:16:17.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-06T00:16:17.783 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:16:17.783 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/docker/daemon.json 2022-03-06T00:16:17.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl restart docker || true 2022-03-06T00:16:17.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl restart docker || true 2022-03-06T00:16:17.852 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl restart docker || true 2022-03-06T00:16:18.947 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-06T00:16:18.957 INFO:tasks.kubeadm:172.21.15.7 is in 172.21.0.0/20 at pos 3846 2022-03-06T00:16:18.973 INFO:tasks.kubeadm:Pod subnet: 10.248.48.0/21 2022-03-06T00:16:18.973 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-06T00:16:18.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-06T00:16:18.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-06T00:16:18.982 DEBUG:teuthology.orchestra.run.smithi066:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-06T00:16:19.002 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-06T00:16:19.003 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:19.003 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-06T00:16:19.010 INFO:teuthology.orchestra.run.smithi017.stderr: 2022-03-06T00:16:19.010 INFO:teuthology.orchestra.run.smithi017.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:19.011 INFO:teuthology.orchestra.run.smithi017.stderr: 2022-03-06T00:16:19.015 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-06T00:16:19.015 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:19.016 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-06T00:16:19.245 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:16:19.246 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:16:19.251 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:16:19.251 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:16:19.252 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:16:19.252 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-06T00:16:19.315 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:16:19.322 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:16:19.325 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-06T00:16:19.652 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:16:19.665 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:16:19.668 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-06T00:16:19.736 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:16:19.744 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:16:19.746 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-06T00:16:20.136 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:16:20.142 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:16:20.147 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-06T00:16:20.372 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 2,292 kB in 1s (1,859 kB/s) 2022-03-06T00:16:20.377 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 2,292 kB in 1s (1,873 kB/s) 2022-03-06T00:16:20.384 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 2,292 kB in 1s (1,857 kB/s) 2022-03-06T00:16:21.231 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-06T00:16:21.241 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-06T00:16:21.251 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2022-03-06T00:16:21.408 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-06T00:16:21.409 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-06T00:16:21.417 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-06T00:16:21.418 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-06T00:16:21.424 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2022-03-06T00:16:21.425 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2022-03-06T00:16:21.500 INFO:teuthology.orchestra.run.smithi066.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:16:21.513 INFO:teuthology.orchestra.run.smithi007.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:16:21.517 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-06T00:16:21.517 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:21.518 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-06T00:16:21.518 INFO:teuthology.orchestra.run.smithi017.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:16:21.530 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-06T00:16:21.530 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:21.531 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-06T00:16:21.533 INFO:teuthology.orchestra.run.smithi017.stderr: 2022-03-06T00:16:21.534 INFO:teuthology.orchestra.run.smithi017.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:21.534 INFO:teuthology.orchestra.run.smithi017.stderr: 2022-03-06T00:16:21.575 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-06T00:16:21.584 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-06T00:16:21.591 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2022-03-06T00:16:21.749 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-06T00:16:21.749 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-06T00:16:21.753 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-06T00:16:21.753 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-06T00:16:21.770 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2022-03-06T00:16:21.770 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2022-03-06T00:16:21.920 INFO:teuthology.orchestra.run.smithi007.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-06T00:16:21.920 INFO:teuthology.orchestra.run.smithi007.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-06T00:16:21.920 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-06T00:16:21.921 INFO:teuthology.orchestra.run.smithi007.stdout: apt-transport-https 2022-03-06T00:16:21.944 INFO:teuthology.orchestra.run.smithi066.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-06T00:16:21.945 INFO:teuthology.orchestra.run.smithi066.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-06T00:16:21.945 INFO:teuthology.orchestra.run.smithi066.stdout:The following NEW packages will be installed: 2022-03-06T00:16:21.945 INFO:teuthology.orchestra.run.smithi066.stdout: apt-transport-https 2022-03-06T00:16:21.963 INFO:teuthology.orchestra.run.smithi017.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-06T00:16:21.964 INFO:teuthology.orchestra.run.smithi017.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-06T00:16:21.964 INFO:teuthology.orchestra.run.smithi017.stdout:The following NEW packages will be installed: 2022-03-06T00:16:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: apt-transport-https 2022-03-06T00:16:22.105 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:16:22.105 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 4,680 B of archives. 2022-03-06T00:16:22.105 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-06T00:16:22.105 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-06T00:16:22.132 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:16:22.132 INFO:teuthology.orchestra.run.smithi066.stdout:Need to get 4,680 B of archives. 2022-03-06T00:16:22.133 INFO:teuthology.orchestra.run.smithi066.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-06T00:16:22.133 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-06T00:16:22.151 INFO:teuthology.orchestra.run.smithi017.stdout:0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:16:22.151 INFO:teuthology.orchestra.run.smithi017.stdout:Need to get 4,680 B of archives. 2022-03-06T00:16:22.152 INFO:teuthology.orchestra.run.smithi017.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-06T00:16:22.152 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-06T00:16:22.332 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:16:22.332 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:16:22.333 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:16:22.356 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:16:22.356 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:16:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:16:22.361 INFO:teuthology.orchestra.run.smithi007.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:16:22.369 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:16:22.370 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:16:22.370 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:16:22.394 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:16:22.395 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:16:22.396 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:16:22.400 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:16:22.401 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:16:22.401 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:16:22.402 INFO:teuthology.orchestra.run.smithi066.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:16:22.412 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-06T00:16:22.422 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:16:22.422 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:16:22.423 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:16:22.428 INFO:teuthology.orchestra.run.smithi017.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:16:22.456 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 4,680 B in 0s (26.3 kB/s) 2022-03-06T00:16:22.477 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 4,680 B in 0s (26.4 kB/s) 2022-03-06T00:16:22.503 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package apt-transport-https. 2022-03-06T00:16:22.541 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-06T00:16:22.544 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-06T00:16:22.546 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package apt-transport-https. 2022-03-06T00:16:22.559 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package apt-transport-https. 2022-03-06T00:16:22.588 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-06T00:16:22.590 INFO:teuthology.orchestra.run.smithi066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-06T00:16:22.593 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-06T00:16:22.605 INFO:teuthology.orchestra.run.smithi017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-06T00:16:22.606 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-06T00:16:22.636 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-06T00:16:22.641 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-06T00:16:22.877 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-06T00:16:22.899 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-06T00:16:22.929 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-06T00:16:23.426 INFO:teuthology.orchestra.run.smithi007.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-06T00:16:23.429 INFO:teuthology.orchestra.run.smithi066.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-06T00:16:23.438 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-06T00:16:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:23.438 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-06T00:16:23.443 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-06T00:16:23.443 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:23.443 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-06T00:16:23.451 INFO:teuthology.orchestra.run.smithi017.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-06T00:16:23.466 INFO:teuthology.orchestra.run.smithi017.stderr: 2022-03-06T00:16:23.466 INFO:teuthology.orchestra.run.smithi017.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:23.467 INFO:teuthology.orchestra.run.smithi017.stderr: 2022-03-06T00:16:23.570 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:16:23.578 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:16:23.604 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-06T00:16:23.679 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:16:23.691 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:16:23.716 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-06T00:16:23.761 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:16:23.775 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:16:23.798 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-06T00:16:23.843 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:16:23.856 INFO:teuthology.orchestra.run.smithi066.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:16:23.881 INFO:teuthology.orchestra.run.smithi017.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-06T00:16:24.232 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-06T00:16:24.362 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-06T00:16:24.397 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-06T00:16:24.397 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-06T00:16:24.493 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 63.3 kB in 1s (70.8 kB/s) 2022-03-06T00:16:24.494 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-06T00:16:24.500 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-06T00:16:24.624 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 63.3 kB in 1s (61.1 kB/s) 2022-03-06T00:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 63.3 kB in 1s (62.7 kB/s) 2022-03-06T00:16:25.371 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-06T00:16:25.492 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-06T00:16:25.505 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2022-03-06T00:16:25.551 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-06T00:16:25.552 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-06T00:16:25.645 INFO:teuthology.orchestra.run.smithi066.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:16:25.659 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-06T00:16:25.659 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:25.660 INFO:teuthology.orchestra.run.smithi066.stderr: 2022-03-06T00:16:25.668 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-06T00:16:25.669 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-06T00:16:25.681 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2022-03-06T00:16:25.682 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2022-03-06T00:16:25.716 INFO:teuthology.orchestra.run.smithi066.stdout:Reading package lists... 2022-03-06T00:16:25.760 INFO:teuthology.orchestra.run.smithi007.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:16:25.775 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-06T00:16:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:25.775 INFO:teuthology.orchestra.run.smithi007.stderr: 2022-03-06T00:16:25.776 INFO:teuthology.orchestra.run.smithi017.stdout:66 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-06T00:16:25.790 INFO:teuthology.orchestra.run.smithi017.stderr: 2022-03-06T00:16:25.791 INFO:teuthology.orchestra.run.smithi017.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-06T00:16:25.791 INFO:teuthology.orchestra.run.smithi017.stderr: 2022-03-06T00:16:25.829 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2022-03-06T00:16:25.847 INFO:teuthology.orchestra.run.smithi017.stdout:Reading package lists... 2022-03-06T00:16:25.896 INFO:teuthology.orchestra.run.smithi066.stdout:Building dependency tree... 2022-03-06T00:16:25.897 INFO:teuthology.orchestra.run.smithi066.stdout:Reading state information... 2022-03-06T00:16:26.006 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2022-03-06T00:16:26.007 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2022-03-06T00:16:26.024 INFO:teuthology.orchestra.run.smithi017.stdout:Building dependency tree... 2022-03-06T00:16:26.025 INFO:teuthology.orchestra.run.smithi017.stdout:Reading state information... 2022-03-06T00:16:26.034 INFO:teuthology.orchestra.run.smithi066.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-06T00:16:26.034 INFO:teuthology.orchestra.run.smithi066.stdout:bridge-utils set to manually installed. 2022-03-06T00:16:26.034 INFO:teuthology.orchestra.run.smithi066.stdout:The following additional packages will be installed: 2022-03-06T00:16:26.035 INFO:teuthology.orchestra.run.smithi066.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-06T00:16:26.036 INFO:teuthology.orchestra.run.smithi066.stdout:Suggested packages: 2022-03-06T00:16:26.036 INFO:teuthology.orchestra.run.smithi066.stdout: nftables 2022-03-06T00:16:26.088 INFO:teuthology.orchestra.run.smithi066.stdout:The following NEW packages will be installed: 2022-03-06T00:16:26.089 INFO:teuthology.orchestra.run.smithi066.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-06T00:16:26.161 INFO:teuthology.orchestra.run.smithi007.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-06T00:16:26.162 INFO:teuthology.orchestra.run.smithi007.stdout:bridge-utils set to manually installed. 2022-03-06T00:16:26.162 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2022-03-06T00:16:26.165 INFO:teuthology.orchestra.run.smithi007.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-06T00:16:26.165 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2022-03-06T00:16:26.166 INFO:teuthology.orchestra.run.smithi007.stdout: nftables 2022-03-06T00:16:26.181 INFO:teuthology.orchestra.run.smithi017.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-06T00:16:26.182 INFO:teuthology.orchestra.run.smithi017.stdout:bridge-utils set to manually installed. 2022-03-06T00:16:26.182 INFO:teuthology.orchestra.run.smithi017.stdout:The following additional packages will be installed: 2022-03-06T00:16:26.182 INFO:teuthology.orchestra.run.smithi017.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-06T00:16:26.183 INFO:teuthology.orchestra.run.smithi017.stdout:Suggested packages: 2022-03-06T00:16:26.184 INFO:teuthology.orchestra.run.smithi017.stdout: nftables 2022-03-06T00:16:26.212 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2022-03-06T00:16:26.213 INFO:teuthology.orchestra.run.smithi007.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-06T00:16:26.232 INFO:teuthology.orchestra.run.smithi017.stdout:The following NEW packages will be installed: 2022-03-06T00:16:26.233 INFO:teuthology.orchestra.run.smithi017.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-06T00:16:26.283 INFO:teuthology.orchestra.run.smithi066.stdout:0 upgraded, 8 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:16:26.283 INFO:teuthology.orchestra.run.smithi066.stdout:Need to get 73.6 MB of archives. 2022-03-06T00:16:26.283 INFO:teuthology.orchestra.run.smithi066.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-06T00:16:26.284 INFO:teuthology.orchestra.run.smithi066.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-06T00:16:26.322 INFO:teuthology.orchestra.run.smithi066.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-06T00:16:26.419 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 8 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:16:26.420 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 73.6 MB of archives. 2022-03-06T00:16:26.420 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-06T00:16:26.420 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-06T00:16:26.447 INFO:teuthology.orchestra.run.smithi017.stdout:0 upgraded, 8 newly installed, 0 to remove and 66 not upgraded. 2022-03-06T00:16:26.448 INFO:teuthology.orchestra.run.smithi017.stdout:Need to get 73.6 MB of archives. 2022-03-06T00:16:26.449 INFO:teuthology.orchestra.run.smithi017.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-06T00:16:26.450 INFO:teuthology.orchestra.run.smithi017.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-06T00:16:26.455 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-06T00:16:26.485 INFO:teuthology.orchestra.run.smithi017.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-06T00:16:26.508 INFO:teuthology.orchestra.run.smithi066.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-06T00:16:26.590 INFO:teuthology.orchestra.run.smithi066.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-06T00:16:26.632 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-06T00:16:26.642 INFO:teuthology.orchestra.run.smithi017.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-06T00:16:26.668 INFO:teuthology.orchestra.run.smithi066.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-06T00:16:26.756 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-06T00:16:26.787 INFO:teuthology.orchestra.run.smithi017.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-06T00:16:26.821 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-06T00:16:27.007 INFO:teuthology.orchestra.run.smithi017.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-06T00:16:27.420 INFO:teuthology.orchestra.run.smithi066.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-06T00:16:27.554 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-06T00:16:27.945 INFO:teuthology.orchestra.run.smithi017.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-06T00:16:28.163 INFO:teuthology.orchestra.run.smithi066.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-06T00:16:28.293 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-06T00:16:28.335 INFO:teuthology.orchestra.run.smithi066.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-06T00:16:28.488 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-06T00:16:28.539 INFO:teuthology.orchestra.run.smithi017.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-06T00:16:28.688 INFO:teuthology.orchestra.run.smithi017.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-06T00:16:28.839 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:16:28.839 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:16:28.839 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:16:28.849 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:16:28.849 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:16:28.849 INFO:teuthology.orchestra.run.smithi066.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:16:28.855 INFO:teuthology.orchestra.run.smithi066.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:16:28.903 INFO:teuthology.orchestra.run.smithi066.stdout:Fetched 73.6 MB in 3s (29.3 MB/s) 2022-03-06T00:16:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:16:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:16:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:16:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:16:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:16:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:16:28.925 INFO:teuthology.orchestra.run.smithi007.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:16:28.976 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 73.6 MB in 2s (29.9 MB/s) 2022-03-06T00:16:28.993 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package conntrack. 2022-03-06T00:16:29.039 INFO:teuthology.orchestra.run.smithi066.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-06T00:16:29.043 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-06T00:16:29.045 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: unable to initialize frontend: Dialog 2022-03-06T00:16:29.045 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-06T00:16:29.045 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: falling back to frontend: Readline 2022-03-06T00:16:29.054 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: unable to initialize frontend: Readline 2022-03-06T00:16:29.054 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-06T00:16:29.055 INFO:teuthology.orchestra.run.smithi017.stderr:debconf: falling back to frontend: Teletype 2022-03-06T00:16:29.060 INFO:teuthology.orchestra.run.smithi017.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-06T00:16:29.070 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package conntrack. 2022-03-06T00:16:29.092 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-06T00:16:29.109 INFO:teuthology.orchestra.run.smithi017.stdout:Fetched 73.6 MB in 3s (28.5 MB/s) 2022-03-06T00:16:29.115 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-06T00:16:29.118 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-06T00:16:29.177 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-06T00:16:29.234 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package conntrack. 2022-03-06T00:16:29.280 INFO:teuthology.orchestra.run.smithi017.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-06T00:16:29.284 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-06T00:16:29.324 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-06T00:16:29.387 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package cri-tools. 2022-03-06T00:16:29.400 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-06T00:16:29.436 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-06T00:16:29.480 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package cri-tools. 2022-03-06T00:16:29.493 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-06T00:16:29.530 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-06T00:16:29.643 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package cri-tools. 2022-03-06T00:16:29.652 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-06T00:16:29.714 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-06T00:16:30.506 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package ebtables. 2022-03-06T00:16:30.520 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-06T00:16:30.579 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-06T00:16:30.774 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package ebtables. 2022-03-06T00:16:30.788 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-06T00:16:30.842 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-06T00:16:30.849 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-06T00:16:30.855 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-06T00:16:30.892 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-06T00:16:30.954 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package ebtables. 2022-03-06T00:16:30.968 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-06T00:16:31.061 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-06T00:16:31.102 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-06T00:16:31.117 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-06T00:16:31.152 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-06T00:16:31.358 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-06T00:16:31.372 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-06T00:16:31.416 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-06T00:16:32.370 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package socat. 2022-03-06T00:16:32.384 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-06T00:16:32.428 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-06T00:16:32.605 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package socat. 2022-03-06T00:16:32.619 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-06T00:16:32.663 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-06T00:16:32.773 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package kubelet. 2022-03-06T00:16:32.786 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-06T00:16:32.822 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-06T00:16:32.910 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package socat. 2022-03-06T00:16:32.923 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-06T00:16:32.959 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-06T00:16:33.092 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubelet. 2022-03-06T00:16:33.106 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-06T00:16:33.150 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-06T00:16:33.356 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package kubelet. 2022-03-06T00:16:33.369 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-06T00:16:33.397 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-06T00:16:35.186 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package kubectl. 2022-03-06T00:16:35.199 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-06T00:16:35.236 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-06T00:16:35.619 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubectl. 2022-03-06T00:16:35.632 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-06T00:16:35.669 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-06T00:16:35.850 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package kubectl. 2022-03-06T00:16:35.864 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-06T00:16:35.908 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-06T00:16:36.430 INFO:teuthology.orchestra.run.smithi066.stdout:Selecting previously unselected package kubeadm. 2022-03-06T00:16:36.446 INFO:teuthology.orchestra.run.smithi066.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-06T00:16:36.488 INFO:teuthology.orchestra.run.smithi066.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-06T00:16:36.922 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package kubeadm. 2022-03-06T00:16:36.935 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-06T00:16:36.980 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-06T00:16:37.246 INFO:teuthology.orchestra.run.smithi017.stdout:Selecting previously unselected package kubeadm. 2022-03-06T00:16:37.261 INFO:teuthology.orchestra.run.smithi017.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-06T00:16:37.321 INFO:teuthology.orchestra.run.smithi017.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-06T00:16:37.593 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-06T00:16:37.719 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-06T00:16:37.879 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-06T00:16:38.132 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-06T00:16:38.245 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-06T00:16:38.381 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-06T00:16:38.600 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-06T00:16:38.642 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-06T00:16:38.752 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-06T00:16:38.769 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-06T00:16:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-06T00:16:38.896 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-06T00:16:39.014 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-06T00:16:39.179 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-06T00:16:39.240 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-06T00:16:39.321 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-06T00:16:39.447 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-06T00:16:39.557 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-06T00:16:39.698 INFO:teuthology.orchestra.run.smithi066.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-06T00:16:39.749 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-06T00:16:39.766 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-06T00:16:39.875 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-06T00:16:40.035 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-06T00:16:40.042 INFO:teuthology.orchestra.run.smithi066.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:16:40.144 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-06T00:16:40.241 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-06T00:16:40.351 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-06T00:16:40.594 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:16:40.820 INFO:teuthology.orchestra.run.smithi017.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-06T00:16:41.506 INFO:teuthology.orchestra.run.smithi017.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-06T00:16:42.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-06T00:16:42.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-06T00:16:42.450 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-06T00:16:54.747 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-06T00:16:55.588 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-06T00:16:56.862 INFO:teuthology.orchestra.run.smithi017.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-06T00:16:58.133 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-06T00:16:58.898 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-06T00:17:00.413 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-06T00:17:00.537 INFO:teuthology.orchestra.run.smithi017.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-06T00:17:01.470 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-06T00:17:02.992 INFO:teuthology.orchestra.run.smithi017.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-06T00:17:04.821 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-06T00:17:06.245 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-06T00:17:06.978 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-06T00:17:07.208 INFO:teuthology.orchestra.run.smithi017.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-06T00:17:08.652 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-06T00:17:08.957 INFO:teuthology.orchestra.run.smithi017.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-06T00:17:11.486 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-06T00:17:13.553 INFO:teuthology.orchestra.run.smithi066.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-06T00:17:15.068 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-06T00:17:15.089 INFO:teuthology.orchestra.run.smithi017.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-06T00:17:17.559 INFO:teuthology.orchestra.run.smithi007.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-06T00:17:17.857 INFO:teuthology.orchestra.run.smithi017.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-06T00:17:17.862 INFO:tasks.kubeadm:Token: abcdef.7o9bb50u3uwnez2a 2022-03-06T00:17:17.863 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): '172.21.15.7', Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): '172.21.15.17', Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): '172.21.15.66'} 2022-03-06T00:17:17.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo kubeadm init --node-name smithi007 --token abcdef.7o9bb50u3uwnez2a --pod-network-cidr 10.248.48.0/21 2022-03-06T00:17:18.054 INFO:teuthology.orchestra.run.smithi007.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-06T00:17:18.054 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] Running pre-flight checks 2022-03-06T00:17:18.465 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-06T00:17:18.465 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-06T00:17:18.466 INFO:teuthology.orchestra.run.smithi007.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-06T00:17:18.778 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-06T00:17:18.898 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "ca" certificate and key 2022-03-06T00:17:18.969 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver" certificate and key 2022-03-06T00:17:18.969 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi007] and IPs [10.96.0.1 172.21.15.7] 2022-03-06T00:17:19.118 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-06T00:17:19.368 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-06T00:17:19.615 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-06T00:17:19.832 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-06T00:17:20.209 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-06T00:17:20.210 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi007] and IPs [172.21.15.7 127.0.0.1 ::1] 2022-03-06T00:17:20.390 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-06T00:17:20.390 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi007] and IPs [172.21.15.7 127.0.0.1 ::1] 2022-03-06T00:17:20.524 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-06T00:17:20.731 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-06T00:17:20.876 INFO:teuthology.orchestra.run.smithi007.stdout:[certs] Generating "sa" key and public key 2022-03-06T00:17:20.876 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-06T00:17:21.111 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-06T00:17:21.226 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-06T00:17:21.575 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-06T00:17:21.888 INFO:teuthology.orchestra.run.smithi007.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-06T00:17:21.903 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-06T00:17:21.904 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-06T00:17:21.905 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-start] Starting the kubelet 2022-03-06T00:17:22.097 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-06T00:17:22.097 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-06T00:17:22.099 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-06T00:17:22.099 INFO:teuthology.orchestra.run.smithi007.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-06T00:17:22.100 INFO:teuthology.orchestra.run.smithi007.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-06T00:17:22.101 INFO:teuthology.orchestra.run.smithi007.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-06T00:17:39.106 INFO:teuthology.orchestra.run.smithi007.stdout:[apiclient] All control plane components are healthy after 17.004118 seconds 2022-03-06T00:17:39.106 INFO:teuthology.orchestra.run.smithi007.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-06T00:17:39.128 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-06T00:17:39.128 INFO:teuthology.orchestra.run.smithi007.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-06T00:17:39.766 INFO:teuthology.orchestra.run.smithi007.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-06T00:17:39.766 INFO:teuthology.orchestra.run.smithi007.stdout:[mark-control-plane] Marking the node smithi007 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-06T00:17:39.767 INFO:teuthology.orchestra.run.smithi007.stdout:[mark-control-plane] Marking the node smithi007 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-06T00:17:40.276 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Using token: abcdef.7o9bb50u3uwnez2a 2022-03-06T00:17:40.276 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-06T00:17:40.350 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-06T00:17:40.370 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-06T00:17:40.378 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-06T00:17:40.402 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-06T00:17:40.411 INFO:teuthology.orchestra.run.smithi007.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-06T00:17:40.488 INFO:teuthology.orchestra.run.smithi007.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-06T00:17:40.904 INFO:teuthology.orchestra.run.smithi007.stdout:[addons] Applied essential addon: CoreDNS 2022-03-06T00:17:41.028 INFO:teuthology.orchestra.run.smithi007.stdout:[addons] Applied essential addon: kube-proxy 2022-03-06T00:17:41.030 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:17:41.031 INFO:teuthology.orchestra.run.smithi007.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-06T00:17:41.031 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:17:41.033 INFO:teuthology.orchestra.run.smithi007.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-06T00:17:41.033 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:17:41.034 INFO:teuthology.orchestra.run.smithi007.stdout: mkdir -p $HOME/.kube 2022-03-06T00:17:41.034 INFO:teuthology.orchestra.run.smithi007.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-06T00:17:41.034 INFO:teuthology.orchestra.run.smithi007.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-06T00:17:41.035 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:17:41.035 INFO:teuthology.orchestra.run.smithi007.stdout:Alternatively, if you are the root user, you can run: 2022-03-06T00:17:41.035 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:17:41.035 INFO:teuthology.orchestra.run.smithi007.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-06T00:17:41.036 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:17:41.036 INFO:teuthology.orchestra.run.smithi007.stdout:You should now deploy a pod network to the cluster. 2022-03-06T00:17:41.036 INFO:teuthology.orchestra.run.smithi007.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-06T00:17:41.036 INFO:teuthology.orchestra.run.smithi007.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-06T00:17:41.036 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:17:41.037 INFO:teuthology.orchestra.run.smithi007.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-06T00:17:41.037 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:17:41.037 INFO:teuthology.orchestra.run.smithi007.stdout:kubeadm join 172.21.15.7:6443 --token abcdef.7o9bb50u3uwnez2a \ 2022-03-06T00:17:41.037 INFO:teuthology.orchestra.run.smithi007.stdout: --discovery-token-ca-cert-hash sha256:3bfb37a3054d7f5ceeebf8efb044b7ab91e057c4aa648d423469cc5deb0273d6 2022-03-06T00:17:41.039 DEBUG:teuthology.orchestra.run.smithi017:> sudo kubeadm join 172.21.15.7:6443 --node-name smithi017 --token abcdef.7o9bb50u3uwnez2a --discovery-token-unsafe-skip-ca-verification 2022-03-06T00:17:41.042 DEBUG:teuthology.orchestra.run.smithi066:> sudo kubeadm join 172.21.15.7:6443 --node-name smithi066 --token abcdef.7o9bb50u3uwnez2a --discovery-token-unsafe-skip-ca-verification 2022-03-06T00:17:41.082 INFO:teuthology.orchestra.run.smithi017.stdout:[preflight] Running pre-flight checks 2022-03-06T00:17:41.083 INFO:teuthology.orchestra.run.smithi066.stdout:[preflight] Running pre-flight checks 2022-03-06T00:17:59.880 INFO:teuthology.orchestra.run.smithi017.stdout:[preflight] Reading configuration from the cluster... 2022-03-06T00:17:59.880 INFO:teuthology.orchestra.run.smithi017.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-06T00:17:59.892 INFO:teuthology.orchestra.run.smithi066.stdout:[preflight] Reading configuration from the cluster... 2022-03-06T00:17:59.893 INFO:teuthology.orchestra.run.smithi066.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-06T00:18:00.239 INFO:teuthology.orchestra.run.smithi017.stderr:W0306 00:18:00.239651 10593 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-06T00:18:00.240 INFO:teuthology.orchestra.run.smithi066.stderr:W0306 00:18:00.240650 10673 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-06T00:18:00.257 INFO:teuthology.orchestra.run.smithi017.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-06T00:18:00.258 INFO:teuthology.orchestra.run.smithi017.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-06T00:18:00.258 INFO:teuthology.orchestra.run.smithi017.stdout:[kubelet-start] Starting the kubelet 2022-03-06T00:18:00.259 INFO:teuthology.orchestra.run.smithi066.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-06T00:18:00.259 INFO:teuthology.orchestra.run.smithi066.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-06T00:18:00.260 INFO:teuthology.orchestra.run.smithi066.stdout:[kubelet-start] Starting the kubelet 2022-03-06T00:18:00.475 INFO:teuthology.orchestra.run.smithi017.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-06T00:18:00.497 INFO:teuthology.orchestra.run.smithi066.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-06T00:18:06.005 INFO:teuthology.orchestra.run.smithi017.stdout: 2022-03-06T00:18:06.006 INFO:teuthology.orchestra.run.smithi017.stdout:This node has joined the cluster: 2022-03-06T00:18:06.006 INFO:teuthology.orchestra.run.smithi017.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-06T00:18:06.006 INFO:teuthology.orchestra.run.smithi017.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-06T00:18:06.007 INFO:teuthology.orchestra.run.smithi017.stdout: 2022-03-06T00:18:06.007 INFO:teuthology.orchestra.run.smithi017.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-06T00:18:06.007 INFO:teuthology.orchestra.run.smithi017.stdout: 2022-03-06T00:18:06.033 INFO:teuthology.orchestra.run.smithi066.stdout: 2022-03-06T00:18:06.034 INFO:teuthology.orchestra.run.smithi066.stdout:This node has joined the cluster: 2022-03-06T00:18:06.034 INFO:teuthology.orchestra.run.smithi066.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-06T00:18:06.034 INFO:teuthology.orchestra.run.smithi066.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-06T00:18:06.034 INFO:teuthology.orchestra.run.smithi066.stdout: 2022-03-06T00:18:06.035 INFO:teuthology.orchestra.run.smithi066.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-06T00:18:06.035 INFO:teuthology.orchestra.run.smithi066.stdout: 2022-03-06T00:18:06.040 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:06.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-06T00:18:06.057 INFO:tasks.kubeadm:Setting up kubectl 2022-03-06T00:18:06.058 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-06T00:18:06.115 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-06T00:18:06.133 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-06T00:18:06.152 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:06.152 DEBUG:teuthology.orchestra.run.smithi007:> dd of=.kube/config 2022-03-06T00:18:06.159 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:06.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/root/.kube/config 2022-03-06T00:18:06.223 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:18:06.224 DEBUG:teuthology.orchestra.run.smithi017:> dd of=.kube/config 2022-03-06T00:18:06.232 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:18:06.232 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/root/.kube/config 2022-03-06T00:18:06.245 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:18:06.245 DEBUG:teuthology.orchestra.run.smithi066:> dd of=.kube/config 2022-03-06T00:18:06.253 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:18:06.253 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/root/.kube/config 2022-03-06T00:18:06.265 DEBUG:teuthology.orchestra.run.smithi007:> kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-03-06T00:18:08.882 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-03-06T00:18:08.931 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-03-06T00:18:09.029 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-03-06T00:18:09.107 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-03-06T00:18:09.189 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-03-06T00:18:09.245 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-03-06T00:18:09.376 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-03-06T00:18:09.382 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-03-06T00:18:09.431 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-03-06T00:18:09.543 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-03-06T00:18:09.572 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-03-06T00:18:09.623 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-03-06T00:18:09.730 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-03-06T00:18:09.782 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-03-06T00:18:09.898 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-03-06T00:18:09.959 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-03-06T00:18:10.065 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-03-06T00:18:10.099 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-03-06T00:18:10.149 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-03-06T00:18:10.276 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-03-06T00:18:10.316 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-03-06T00:18:10.434 INFO:teuthology.orchestra.run.smithi007.stdout:namespace/tigera-operator created 2022-03-06T00:18:10.441 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-06T00:18:10.465 INFO:teuthology.orchestra.run.smithi007.stdout:podsecuritypolicy.policy/tigera-operator created 2022-03-06T00:18:10.496 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/tigera-operator created 2022-03-06T00:18:10.608 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-03-06T00:18:10.632 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-03-06T00:18:10.647 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps/tigera-operator created 2022-03-06T00:18:10.654 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f - 2022-03-06T00:18:13.638 INFO:teuthology.orchestra.run.smithi007.stdout:installation.operator.tigera.io/default created 2022-03-06T00:18:13.645 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-06T00:18:13.645 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:13.646 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:13.652 INFO:tasks.kubeadm:Scratch devices on smithi007: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:18:13.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-06T00:18:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-06T00:18:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-06T00:18:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134305 s, 781 MB/s 2022-03-06T00:18:13.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-06T00:18:13.701 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-06T00:18:13.701 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-06T00:18:13.701 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129014 s, 813 MB/s 2022-03-06T00:18:13.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-06T00:18:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-06T00:18:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-06T00:18:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129223 s, 811 MB/s 2022-03-06T00:18:13.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-06T00:18:13.796 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records in 2022-03-06T00:18:13.797 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2022-03-06T00:18:13.797 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131554 s, 797 MB/s 2022-03-06T00:18:13.798 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:18:13.799 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:13.808 INFO:tasks.kubeadm:Scratch devices on smithi017: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:18:13.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-06T00:18:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records in 2022-03-06T00:18:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2022-03-06T00:18:13.836 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0154265 s, 680 MB/s 2022-03-06T00:18:13.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-06T00:18:13.912 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records in 2022-03-06T00:18:13.912 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2022-03-06T00:18:13.912 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140537 s, 746 MB/s 2022-03-06T00:18:13.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-06T00:18:13.936 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records in 2022-03-06T00:18:13.936 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2022-03-06T00:18:13.936 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012524 s, 837 MB/s 2022-03-06T00:18:13.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-06T00:18:13.965 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records in 2022-03-06T00:18:13.965 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2022-03-06T00:18:13.965 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0136886 s, 766 MB/s 2022-03-06T00:18:13.966 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:18:13.967 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:13.975 INFO:tasks.kubeadm:Scratch devices on smithi066: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:18:13.975 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-06T00:18:14.047 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records in 2022-03-06T00:18:14.048 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2022-03-06T00:18:14.048 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015652 s, 670 MB/s 2022-03-06T00:18:14.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-06T00:18:14.081 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records in 2022-03-06T00:18:14.082 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2022-03-06T00:18:14.082 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0177951 s, 589 MB/s 2022-03-06T00:18:14.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-06T00:18:14.114 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records in 2022-03-06T00:18:14.114 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2022-03-06T00:18:14.115 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0173918 s, 603 MB/s 2022-03-06T00:18:14.116 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-06T00:18:14.148 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records in 2022-03-06T00:18:14.149 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2022-03-06T00:18:14.149 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0170297 s, 616 MB/s 2022-03-06T00:18:14.169 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-06T00:18:14.169 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi007-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi007 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi017-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi017 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi017-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi017 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi017-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi017 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi017-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi017 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi066-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi066 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi066-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi066 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi066-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi066 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi066-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi066 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-06T00:18:14.170 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f - 2022-03-06T00:18:15.048 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-1 created 2022-03-06T00:18:15.057 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-2 created 2022-03-06T00:18:15.137 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-3 created 2022-03-06T00:18:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi007-lv-4 created 2022-03-06T00:18:15.150 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi017-lv-1 created 2022-03-06T00:18:15.158 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi017-lv-2 created 2022-03-06T00:18:15.178 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi017-lv-3 created 2022-03-06T00:18:15.183 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi017-lv-4 created 2022-03-06T00:18:15.191 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi066-lv-1 created 2022-03-06T00:18:15.212 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi066-lv-2 created 2022-03-06T00:18:15.217 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi066-lv-3 created 2022-03-06T00:18:15.265 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi066-lv-4 created 2022-03-06T00:18:15.268 INFO:teuthology.orchestra.run.smithi007.stdout:storageclass.storage.k8s.io/scratch created 2022-03-06T00:18:15.273 DEBUG:teuthology.orchestra.run.smithi007:> kubectl taint node smithi007 node-role.kubernetes.io/master- || true 2022-03-06T00:18:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:node/smithi007 untainted 2022-03-06T00:18:15.341 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-06T00:18:15.341 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-06T00:18:15.369 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-06T00:18:15.370 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:15.370 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:15.379 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:18:15.379 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-06T00:18:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-06T00:18:15.386 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:15.387 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-06T00:18:15.387 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:15.387 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-06 00:18:13.679015044 +0000 2022-03-06T00:18:15.387 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-06 00:18:13.679015044 +0000 2022-03-06T00:18:15.387 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-06 00:18:13.679015044 +0000 2022-03-06T00:18:15.388 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-06T00:18:15.397 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-06T00:18:15.397 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-06T00:18:15.397 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000334624 s, 1.5 MB/s 2022-03-06T00:18:15.401 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-06T00:18:15.407 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-06T00:18:15.408 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:15.408 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-06T00:18:15.408 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:15.408 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-06 00:18:13.703015043 +0000 2022-03-06T00:18:15.409 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-06 00:18:13.703015043 +0000 2022-03-06T00:18:15.409 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-06 00:18:13.703015043 +0000 2022-03-06T00:18:15.409 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-06T00:18:15.416 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-06T00:18:15.416 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-06T00:18:15.417 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000280703 s, 1.8 MB/s 2022-03-06T00:18:15.420 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-06T00:18:15.426 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-06T00:18:15.427 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:15.427 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-03-06T00:18:15.427 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:15.427 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-06 00:18:13.771015042 +0000 2022-03-06T00:18:15.428 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-06 00:18:13.771015042 +0000 2022-03-06T00:18:15.428 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-06 00:18:13.771015042 +0000 2022-03-06T00:18:15.429 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-06T00:18:15.434 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-06T00:18:15.434 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-06T00:18:15.435 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.00023881 s, 2.1 MB/s 2022-03-06T00:18:15.439 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-06T00:18:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-06T00:18:15.490 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:15.490 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-06T00:18:15.491 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:15.491 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2022-03-06 00:18:13.799015041 +0000 2022-03-06T00:18:15.491 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2022-03-06 00:18:13.799015041 +0000 2022-03-06T00:18:15.491 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2022-03-06 00:18:13.799015041 +0000 2022-03-06T00:18:15.491 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2022-03-06T00:18:15.495 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2022-03-06T00:18:15.496 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2022-03-06T00:18:15.496 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000232822 s, 2.2 MB/s 2022-03-06T00:18:15.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-06T00:18:15.584 INFO:teuthology.orchestra.run.smithi007.stdout:loop 2022-03-06T00:18:15.586 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_1... 2022-03-06T00:18:15.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-06T00:18:15.608 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-06T00:18:15.633 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_11 2022-03-06T00:18:15.659 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_2... 2022-03-06T00:18:15.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-06T00:18:15.728 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-06T00:18:15.754 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_21 2022-03-06T00:18:15.779 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_3... 2022-03-06T00:18:15.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-06T00:18:15.803 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-06T00:18:15.829 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_31 2022-03-06T00:18:15.853 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_4... 2022-03-06T00:18:15.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-06T00:18:15.878 INFO:teuthology.orchestra.run.smithi007.stdout:1 2022-03-06T00:18:15.904 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_41 2022-03-06T00:18:15.928 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:15.928 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:15.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo nvme list 2022-03-06T00:18:15.994 INFO:teuthology.orchestra.run.smithi007.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-06T00:18:15.994 INFO:teuthology.orchestra.run.smithi007.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-06T00:18:15.994 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme0n1 PHMB751300GE480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-06T00:18:15.995 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme1n1 8e9bde6914d87c0d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:15.995 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme2n1 3079f1da5323c210 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:15.995 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme3n1 478ea8a88b796241 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:15.996 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme4n1 48fdde6bbd98c340 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:15.997 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-06T00:18:15.997 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:15.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/scratch_devs 2022-03-06T00:18:16.010 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:18:16.010 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:16.019 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:18:16.019 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-06T00:18:16.068 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-06T00:18:16.069 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:16.069 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-03-06T00:18:16.070 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:16.070 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2022-03-06 00:18:13.836397302 +0000 2022-03-06T00:18:16.071 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2022-03-06 00:18:13.836397302 +0000 2022-03-06T00:18:16.071 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2022-03-06 00:18:13.836397302 +0000 2022-03-06T00:18:16.071 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2022-03-06T00:18:16.079 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2022-03-06T00:18:16.079 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2022-03-06T00:18:16.080 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.0002938 s, 1.7 MB/s 2022-03-06T00:18:16.083 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-06T00:18:16.090 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-06T00:18:16.091 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:16.091 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-03-06T00:18:16.091 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:16.091 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2022-03-06 00:18:13.912397300 +0000 2022-03-06T00:18:16.092 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2022-03-06 00:18:13.912397300 +0000 2022-03-06T00:18:16.092 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2022-03-06 00:18:13.912397300 +0000 2022-03-06T00:18:16.092 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2022-03-06T00:18:16.099 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2022-03-06T00:18:16.099 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2022-03-06T00:18:16.099 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000300692 s, 1.7 MB/s 2022-03-06T00:18:16.103 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-06T00:18:16.109 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-06T00:18:16.110 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:16.110 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-03-06T00:18:16.110 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:16.110 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2022-03-06 00:18:13.940397299 +0000 2022-03-06T00:18:16.111 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2022-03-06 00:18:13.940397299 +0000 2022-03-06T00:18:16.111 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2022-03-06 00:18:13.940397299 +0000 2022-03-06T00:18:16.111 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2022-03-06T00:18:16.118 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2022-03-06T00:18:16.118 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2022-03-06T00:18:16.118 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000260268 s, 2.0 MB/s 2022-03-06T00:18:16.122 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-06T00:18:16.172 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-06T00:18:16.172 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:16.172 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 520 Links: 1 2022-03-06T00:18:16.173 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:16.173 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2022-03-06 00:18:13.964397299 +0000 2022-03-06T00:18:16.173 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2022-03-06 00:18:13.964397299 +0000 2022-03-06T00:18:16.173 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2022-03-06 00:18:13.964397299 +0000 2022-03-06T00:18:16.174 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: - 2022-03-06T00:18:16.181 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2022-03-06T00:18:16.181 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2022-03-06T00:18:16.182 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000281858 s, 1.8 MB/s 2022-03-06T00:18:16.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-06T00:18:16.258 INFO:teuthology.orchestra.run.smithi017.stdout:loop 2022-03-06T00:18:16.260 INFO:tasks.nvme_loop:Connecting nvme_loop smithi017:/dev/vg_nvme/lv_1... 2022-03-06T00:18:16.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-06T00:18:16.281 INFO:teuthology.orchestra.run.smithi017.stdout:1 2022-03-06T00:18:16.308 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/vg_nvme/lv_11 2022-03-06T00:18:16.332 INFO:tasks.nvme_loop:Connecting nvme_loop smithi017:/dev/vg_nvme/lv_2... 2022-03-06T00:18:16.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-06T00:18:16.358 INFO:teuthology.orchestra.run.smithi017.stdout:1 2022-03-06T00:18:16.381 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/vg_nvme/lv_21 2022-03-06T00:18:16.402 INFO:tasks.nvme_loop:Connecting nvme_loop smithi017:/dev/vg_nvme/lv_3... 2022-03-06T00:18:16.402 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-06T00:18:16.466 INFO:teuthology.orchestra.run.smithi017.stdout:1 2022-03-06T00:18:16.491 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/vg_nvme/lv_31 2022-03-06T00:18:16.513 INFO:tasks.nvme_loop:Connecting nvme_loop smithi017:/dev/vg_nvme/lv_4... 2022-03-06T00:18:16.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-06T00:18:16.537 INFO:teuthology.orchestra.run.smithi017.stdout:1 2022-03-06T00:18:16.560 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/vg_nvme/lv_41 2022-03-06T00:18:16.583 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:18:16.583 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:16.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo nvme list 2022-03-06T00:18:16.648 INFO:teuthology.orchestra.run.smithi017.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-06T00:18:16.648 INFO:teuthology.orchestra.run.smithi017.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-06T00:18:16.649 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme0n1 PHMB7473003Q480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-06T00:18:16.649 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme1n1 bc0ebf35c4b4570a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:16.649 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme2n1 8ab343d9afbd9b1b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:16.650 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme3n1 81d4c8297746a29b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:16.650 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme4n1 da3721cdea0f777c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:16.651 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-06T00:18:16.651 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:18:16.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/scratch_devs 2022-03-06T00:18:16.666 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:18:16.666 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:16.675 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-06T00:18:16.675 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-06T00:18:16.682 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-06T00:18:16.682 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:16.683 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 498 Links: 1 2022-03-06T00:18:16.683 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:16.683 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2022-03-06 00:18:14.047291697 +0000 2022-03-06T00:18:16.683 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2022-03-06 00:18:14.047291697 +0000 2022-03-06T00:18:16.684 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2022-03-06 00:18:14.047291697 +0000 2022-03-06T00:18:16.684 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2022-03-06T00:18:16.690 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2022-03-06T00:18:16.690 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2022-03-06T00:18:16.690 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000505498 s, 1.0 MB/s 2022-03-06T00:18:16.694 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-06T00:18:16.701 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-06T00:18:16.701 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:16.701 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-03-06T00:18:16.702 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:16.702 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2022-03-06 00:18:14.083291697 +0000 2022-03-06T00:18:16.703 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2022-03-06 00:18:14.083291697 +0000 2022-03-06T00:18:16.703 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2022-03-06 00:18:14.083291697 +0000 2022-03-06T00:18:16.703 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2022-03-06T00:18:16.708 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2022-03-06T00:18:16.708 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2022-03-06T00:18:16.709 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000452088 s, 1.1 MB/s 2022-03-06T00:18:16.711 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-06T00:18:16.759 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-06T00:18:16.760 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:16.760 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 510 Links: 1 2022-03-06T00:18:16.760 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:16.760 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2022-03-06 00:18:14.115291698 +0000 2022-03-06T00:18:16.761 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2022-03-06 00:18:14.115291698 +0000 2022-03-06T00:18:16.761 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2022-03-06 00:18:14.115291698 +0000 2022-03-06T00:18:16.761 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2022-03-06T00:18:16.767 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2022-03-06T00:18:16.767 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2022-03-06T00:18:16.768 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000455911 s, 1.1 MB/s 2022-03-06T00:18:16.771 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-06T00:18:16.775 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-06T00:18:16.776 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-06T00:18:16.776 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 515 Links: 1 2022-03-06T00:18:16.776 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-06T00:18:16.777 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2022-03-06 00:18:14.147291699 +0000 2022-03-06T00:18:16.777 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2022-03-06 00:18:14.147291699 +0000 2022-03-06T00:18:16.777 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2022-03-06 00:18:14.147291699 +0000 2022-03-06T00:18:16.777 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: - 2022-03-06T00:18:16.783 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2022-03-06T00:18:16.783 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2022-03-06T00:18:16.784 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000441379 s, 1.2 MB/s 2022-03-06T00:18:16.787 DEBUG:teuthology.orchestra.run.smithi066:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-06T00:18:16.925 INFO:teuthology.orchestra.run.smithi066.stdout:loop 2022-03-06T00:18:16.927 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_1... 2022-03-06T00:18:16.927 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-06T00:18:16.948 INFO:teuthology.orchestra.run.smithi066.stdout:1 2022-03-06T00:18:16.977 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_11 2022-03-06T00:18:17.002 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_2... 2022-03-06T00:18:17.003 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-06T00:18:17.024 INFO:teuthology.orchestra.run.smithi066.stdout:1 2022-03-06T00:18:17.046 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_21 2022-03-06T00:18:17.064 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_3... 2022-03-06T00:18:17.065 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-06T00:18:17.130 INFO:teuthology.orchestra.run.smithi066.stdout:1 2022-03-06T00:18:17.160 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_31 2022-03-06T00:18:17.185 INFO:tasks.nvme_loop:Connecting nvme_loop smithi066:/dev/vg_nvme/lv_4... 2022-03-06T00:18:17.185 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-06T00:18:17.207 INFO:teuthology.orchestra.run.smithi066.stdout:1 2022-03-06T00:18:17.237 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/vg_nvme/lv_41 2022-03-06T00:18:17.257 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:18:17.258 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:18:17.263 DEBUG:teuthology.orchestra.run.smithi066:> sudo nvme list 2022-03-06T00:18:17.318 INFO:teuthology.orchestra.run.smithi066.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-06T00:18:17.319 INFO:teuthology.orchestra.run.smithi066.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-06T00:18:17.319 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme0n1 CVFT53430091400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-06T00:18:17.319 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme1n1 c261733931f934af Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:17.319 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme2n1 55949e9116e2dc60 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:17.320 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme3n1 185d964af7f0a2fb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:17.320 INFO:teuthology.orchestra.run.smithi066.stdout:/dev/nvme4n1 f03e83e3ed5afde9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-06T00:18:17.321 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-06T00:18:17.321 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:18:17.322 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/scratch_devs 2022-03-06T00:18:17.335 INFO:teuthology.run_tasks:Running task rook... 2022-03-06T00:18:17.439 INFO:tasks.rook:Rook start 2022-03-06T00:18:17.440 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'd0dc2fbbf3f64a2d513e2355b245d0ea9728ff97', 'rook_image': 'rook/ceph:master'} 2022-03-06T00:18:17.440 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 2022-03-06T00:18:17.441 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-06T00:18:17.441 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-06T00:18:17.451 INFO:teuthology.orchestra.run.smithi007.stderr:Cloning into 'rook'... 2022-03-06T00:18:21.677 INFO:tasks.rook:/home/teuthworker 2022-03-06T00:18:21.678 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/sbin:/usr/sbin ip addr show 2022-03-06T00:18:21.686 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-06T00:18:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-06T00:18:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2022-03-06T00:18:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-06T00:18:21.687 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2022-03-06T00:18:21.688 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-06T00:18:21.688 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-06T00:18:21.688 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2022-03-06T00:18:21.688 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2022-03-06T00:18:21.689 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-06T00:18:21.689 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2022-03-06T00:18:21.689 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2022-03-06T00:18:21.689 INFO:teuthology.orchestra.run.smithi007.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-06T00:18:21.690 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2022-03-06T00:18:21.690 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2022-03-06T00:18:21.690 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-03-06T00:18:21.690 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42793sec preferred_lft 42793sec 2022-03-06T00:18:21.691 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2022-03-06T00:18:21.691 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-06T00:18:21.691 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-06T00:18:21.691 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2022-03-06T00:18:21.692 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2022-03-06T00:18:21.692 INFO:teuthology.orchestra.run.smithi007.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-06T00:18:21.692 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 02:42:75:b2:1e:3f brd ff:ff:ff:ff:ff:ff 2022-03-06T00:18:21.692 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-06T00:18:21.693 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2022-03-06T00:18:21.694 DEBUG:teuthology.orchestra.run.smithi007:> which systemctl 2022-03-06T00:18:21.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/systemctl 2022-03-06T00:18:21.738 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-06T00:18:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-06T00:18:21.744 DEBUG:teuthology.orchestra.run.smithi007:> true 2022-03-06T00:18:21.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo lsb_release -is 2022-03-06T00:18:21.823 INFO:teuthology.orchestra.run.smithi007.stdout:Ubuntu 2022-03-06T00:18:21.828 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-06T00:18:21.829 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-06T00:18:21.842 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:18:21.857 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:21.857 DEBUG:teuthology.orchestra.run.smithi007:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-06T00:18:21.867 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-06T00:18:21.908 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:18:21.908 DEBUG:teuthology.orchestra.run.smithi007:> dd of=operator.yaml 2022-03-06T00:18:21.922 INFO:tasks.rook:Deploying operator 2022-03-06T00:18:21.932 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:18:22.046 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:18:22.061 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-06T00:18:22.338 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-06T00:18:22.362 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-06T00:18:22.418 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-06T00:18:22.435 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-06T00:18:22.614 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-06T00:18:22.639 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-06T00:18:22.770 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-06T00:18:22.797 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-06T00:18:22.845 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-06T00:18:22.986 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-06T00:18:23.056 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-06T00:18:23.070 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-06T00:18:23.140 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-06T00:18:23.164 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-06T00:18:23.193 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-06T00:18:23.212 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-06T00:18:23.238 INFO:teuthology.orchestra.run.smithi007.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-06T00:18:23.339 INFO:teuthology.orchestra.run.smithi007.stdout:namespace/rook-ceph created 2022-03-06T00:18:23.348 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-06T00:18:23.373 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-06T00:18:23.387 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-06T00:18:23.389 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-06T00:18:23.412 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-06T00:18:23.464 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-06T00:18:23.470 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-06T00:18:23.529 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-06T00:18:23.539 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-06T00:18:23.548 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-06T00:18:23.556 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-06T00:18:23.614 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-06T00:18:23.617 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-06T00:18:23.622 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-06T00:18:23.631 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-06T00:18:23.639 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-06T00:18:23.648 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-06T00:18:23.656 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-06T00:18:23.679 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-06T00:18:23.689 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-06T00:18:23.697 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-06T00:18:23.708 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-06T00:18:23.781 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-06T00:18:23.783 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-06T00:18:23.792 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-06T00:18:23.797 INFO:teuthology.orchestra.run.smithi007.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-06T00:18:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-06T00:18:23.806 INFO:teuthology.orchestra.run.smithi007.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-06T00:18:23.814 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-06T00:18:23.846 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-06T00:18:23.865 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-06T00:18:23.905 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-06T00:18:23.909 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-06T00:18:23.923 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-06T00:18:23.940 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-06T00:18:23.956 INFO:teuthology.orchestra.run.smithi007.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-06T00:18:23.994 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-06T00:18:23.998 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-06T00:18:24.055 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-06T00:18:24.058 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-06T00:18:24.065 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-06T00:18:24.073 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-06T00:18:24.096 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-06T00:18:24.111 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-06T00:18:24.120 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-06T00:18:24.126 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-06T00:18:24.131 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-06T00:18:24.139 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-06T00:18:24.190 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-06T00:18:24.194 INFO:teuthology.orchestra.run.smithi007.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-06T00:18:24.221 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-06T00:18:24.231 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-mgr created 2022-03-06T00:18:24.239 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-osd created 2022-03-06T00:18:24.265 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-06T00:18:24.273 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-ceph-system created 2022-03-06T00:18:24.330 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-06T00:18:24.334 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-06T00:18:24.340 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-06T00:18:24.369 INFO:teuthology.orchestra.run.smithi007.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-06T00:18:24.396 INFO:teuthology.orchestra.run.smithi007.stdout:configmap/rook-ceph-operator-config created 2022-03-06T00:18:24.414 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps/rook-ceph-operator created 2022-03-06T00:18:24.422 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:18:24.508 INFO:teuthology.orchestra.run.smithi007.stderr:No resources found in rook-ceph namespace. 2022-03-06T00:18:34.514 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:18:34.719 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:18:34.719 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-xbdg5 0/1 Pending 0 10s 2022-03-06T00:18:44.722 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:18:44.784 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:18:44.784 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-xbdg5 0/1 Pending 0 20s 2022-03-06T00:18:54.788 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:18:54.861 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:18:54.861 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-xbdg5 0/1 ContainerCreating 0 30s 2022-03-06T00:19:04.863 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:19:04.934 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:04.934 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-xbdg5 0/1 ContainerCreating 0 40s 2022-03-06T00:19:14.936 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:19:15.009 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:15.009 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-xbdg5 0/1 ContainerCreating 0 51s 2022-03-06T00:19:25.013 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:19:25.083 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:25.084 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-xbdg5 0/1 ContainerCreating 0 61s 2022-03-06T00:19:35.087 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:19:35.153 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:35.153 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-xbdg5 0/1 ContainerCreating 0 71s 2022-03-06T00:19:45.156 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-06T00:19:45.222 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:45.222 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-operator-75dd789779-xbdg5 1/1 Running 0 81s 2022-03-06T00:19:45.225 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-xbdg5 2022-03-06T00:19:45.235 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:19:45.236 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:45.244 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:19:45.244 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:45.253 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:19:45.254 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2022-03-06T00:19:45.263 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-06T00:19:45.264 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-06T00:19:45.264 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-06T00:19:45.264 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-06T00:19:45.265 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-06T00:19:45.265 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-06T00:19:45.265 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-06T00:19:45.266 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-06T00:19:45.269 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f - 2022-03-06T00:19:45.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:36.830431 I | rookcmd: starting Rook v1.8.0-alpha.0.373.g507a2e336 with arguments '/usr/local/bin/rook ceph operator' 2022-03-06T00:19:45.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:36.830569 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-06T00:19:45.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:36.830574 I | cephcmd: starting Rook-Ceph operator 2022-03-06T00:19:45.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:36.988910 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-06T00:19:45.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:36.998213 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:19:45.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:36.998239 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:19:45.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:36.998309 I | operator: setting up schemes 2022-03-06T00:19:45.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.000627 I | operator: setting up the controller-runtime manager 2022-03-06T00:19:45.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.307229 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:19:45.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310517 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:19:45.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310584 I | ceph-cluster-controller: successfully started 2022-03-06T00:19:45.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310688 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:19:45.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310730 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:19:45.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310768 I | ceph-block-pool-controller: successfully started 2022-03-06T00:19:45.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310818 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:19:45.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310854 I | ceph-object-realm-controller: successfully started 2022-03-06T00:19:45.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310884 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:19:45.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.310912 I | ceph-object-zone-controller: successfully started 2022-03-06T00:19:45.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311093 I | ceph-object-controller: successfully started 2022-03-06T00:19:45.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311134 I | ceph-file-controller: successfully started 2022-03-06T00:19:45.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311172 I | ceph-nfs-controller: successfully started 2022-03-06T00:19:45.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311207 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:19:45.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311233 I | ceph-client-controller: successfully started 2022-03-06T00:19:45.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311256 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:19:45.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311281 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:19:45.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311301 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:19:45.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311318 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:19:45.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311333 I | ceph-bucket-topic: successfully started 2022-03-06T00:19:45.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311348 I | ceph-bucket-notification: successfully started 2022-03-06T00:19:45.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311361 I | ceph-bucket-notification: successfully started 2022-03-06T00:19:45.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.311380 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:19:45.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.312528 I | operator: starting the controller-runtime manager 2022-03-06T00:19:45.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.414747 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:19:45.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.414798 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:19:45.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.414828 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:19:45.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.419893 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:19:45.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.424796 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-06T00:19:45.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.424827 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-06T00:19:45.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.427484 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-06T00:19:45.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.435084 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-06T00:19:45.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.435108 I | ceph-csi: CSI CephFS driver disabled 2022-03-06T00:19:45.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.435117 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-06T00:19:45.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.437338 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-06T00:19:45.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:37.443709 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-06T00:19:46.244 INFO:teuthology.orchestra.run.smithi007.stdout:configmap/rook-config-override created 2022-03-06T00:19:46.249 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-06T00:19:46.250 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:19:46.251 DEBUG:teuthology.orchestra.run.smithi007:> dd of=cluster.yaml 2022-03-06T00:19:46.711 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f cluster.yaml 2022-03-06T00:19:49.238 INFO:teuthology.orchestra.run.smithi007.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-06T00:19:49.245 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.253167 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-06T00:19:49.251 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:19:49.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.264945 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:19:49.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.266728 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-06T00:19:49.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.269168 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-06T00:19:49.265 DEBUG:teuthology.orchestra.run.smithi007:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-06T00:19:49.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.290056 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:19:49.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308476 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-06T00:19:49.302 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308503 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-06T00:19:49.302 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308510 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-06T00:19:49.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308516 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-06T00:19:49.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308522 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-06T00:19:49.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308529 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-06T00:19:49.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308536 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-06T00:19:49.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308543 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-06T00:19:49.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308550 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-06T00:19:49.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308558 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-06T00:19:49.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308564 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-06T00:19:49.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308571 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-06T00:19:49.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308578 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-06T00:19:49.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308584 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-06T00:19:49.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308590 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-06T00:19:49.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308597 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-06T00:19:49.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308676 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:19:49.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:19:49.308690 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:19:49.600 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps/rook-ceph-tools created 2022-03-06T00:19:49.603 INFO:tasks.rook:Waiting for tools container to start 2022-03-06T00:19:49.604 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:19:49.763 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:49.764 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 Pending 0 0s 2022-03-06T00:19:54.768 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:19:54.832 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:54.833 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 5s 2022-03-06T00:19:59.836 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:19:59.908 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:19:59.909 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 10s 2022-03-06T00:20:04.912 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:04.982 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:04.983 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 15s 2022-03-06T00:20:09.985 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:10.065 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:10.066 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 21s 2022-03-06T00:20:15.068 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:15.140 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 26s 2022-03-06T00:20:20.143 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:20.211 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:20.211 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 31s 2022-03-06T00:20:25.215 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:25.288 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:25.289 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 36s 2022-03-06T00:20:30.293 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:30.362 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:30.362 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 41s 2022-03-06T00:20:35.366 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:35.434 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:35.434 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 46s 2022-03-06T00:20:40.437 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:40.505 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:40.506 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 51s 2022-03-06T00:20:45.508 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:45.571 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:45.571 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 56s 2022-03-06T00:20:50.573 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:50.641 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:50.642 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 61s 2022-03-06T00:20:55.645 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:20:55.713 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:20:55.713 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 66s 2022-03-06T00:21:00.716 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:00.784 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:00.785 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 71s 2022-03-06T00:21:05.787 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:05.853 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:05.854 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 76s 2022-03-06T00:21:07.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.477209 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:21:07.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.477239 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:21:07.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.477256 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:21:07.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.529609 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:21:07.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.574375 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-06 00:21:07.554224742 +0000 UTC m=+90.745838840 LastTransitionTime:2022-03-06 00:21:07.554224585 +0000 UTC m=+90.745838694}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:21:07.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.588575 I | op-mon: start running mons 2022-03-06T00:21:07.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.670195 I | op-mon: creating mon secrets for a new cluster 2022-03-06T00:21:07.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.715785 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-06T00:21:07.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:07.718668 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-06T00:21:08.428 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:08.437031 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:21:08.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:08.437244 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:21:10.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.038133 I | op-mon: parsing mon endpoints: 2022-03-06T00:21:10.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.038166 W | op-mon: ignoring invalid monitor 2022-03-06T00:21:10.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.038209 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-06T00:21:10.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.038227 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:21:10.034 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.040402 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-06T00:21:10.035 INFO:tasks.rook.operator.smithi007.stdout:I0306 00:21:10.040507 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:21:10.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.437675 I | op-mon: targeting the mon count 3 2022-03-06T00:21:10.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.452031 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-06T00:21:10.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.490631 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-06T00:21:10.513 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:10.522097 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-06T00:21:10.857 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:10.931 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:10.931 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 81s 2022-03-06T00:21:11.231 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.239885 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi066 2022-03-06T00:21:11.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.239922 I | op-mon: mon a assigned to node smithi066 2022-03-06T00:21:11.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.438801 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi017 2022-03-06T00:21:11.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.438837 I | op-mon: mon b assigned to node smithi017 2022-03-06T00:21:11.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.638566 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi007 2022-03-06T00:21:11.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.638583 I | op-mon: mon c assigned to node smithi007 2022-03-06T00:21:11.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.644256 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-06T00:21:11.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.653522 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-06T00:21:11.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.799014 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-06T00:21:11.965 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:11.973778 I | op-mon: creating mon a 2022-03-06T00:21:12.141 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:12.149864 I | op-mon: mon "a" endpoint is [v2:10.102.190.159:3300,v1:10.102.190.159:6789] 2022-03-06T00:21:12.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:12.471955 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789"]}] data:a=10.102.190.159:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:-1] 2022-03-06T00:21:12.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:12.472030 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-06T00:21:12.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:12.472092 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-06T00:21:13.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:13.035783 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:21:13.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:13.036071 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:21:13.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:13.452503 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-06T00:21:13.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:13.702196 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-06T00:21:14.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:14.437698 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789"]}] data:a=10.102.190.159:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:0] 2022-03-06T00:21:14.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:14.437734 I | op-mon: waiting for mon quorum with [a] 2022-03-06T00:21:14.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:14.639801 I | op-mon: mon a is not yet running 2022-03-06T00:21:14.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:14.639837 I | op-mon: mons running: [] 2022-03-06T00:21:15.934 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:16.004 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:16.005 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 87s 2022-03-06T00:21:21.007 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:21.082 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:21.083 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 92s 2022-03-06T00:21:26.085 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:26.158 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:26.158 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 97s 2022-03-06T00:21:31.160 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:31.223 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:31.223 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 102s 2022-03-06T00:21:34.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:34.898302 I | op-mon: mons running: [a] 2022-03-06T00:21:35.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:35.129674 I | op-mon: Monitors in quorum: [a] 2022-03-06T00:21:35.121 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:35.129704 I | op-mon: mons created: 1 2022-03-06T00:21:35.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:35.360167 I | op-mon: waiting for mon quorum with [a] 2022-03-06T00:21:35.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:35.370370 I | op-mon: mons running: [a] 2022-03-06T00:21:35.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:35.594504 I | op-mon: Monitors in quorum: [a] 2022-03-06T00:21:35.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:35.594565 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:21:35.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:35.885374 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:21:35.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:35.885408 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:21:36.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:36.177338 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:21:36.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:36.177363 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:21:36.226 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:36.317 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:36.317 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 107s 2022-03-06T00:21:36.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:36.468903 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:21:36.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:36.468943 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:21:36.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:36.756003 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:21:36.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:36.756034 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:21:36.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:36.971319 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:21:36.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:36.971362 I | op-mon: creating mon b 2022-03-06T00:21:38.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:38.153705 I | op-mon: mon "a" endpoint is [v2:10.102.190.159:3300,v1:10.102.190.159:6789] 2022-03-06T00:21:38.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:38.595456 I | op-mon: mon "b" endpoint is [v2:10.98.23.111:3300,v1:10.98.23.111:6789] 2022-03-06T00:21:39.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:39.089899 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.23.111:6789","10.102.190.159:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:0] 2022-03-06T00:21:39.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:39.724853 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:21:39.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:39.725191 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:21:40.053 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:40.062484 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:21:40.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:40.071161 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:21:40.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:40.071195 I | op-mon: waiting for mon quorum with [a b] 2022-03-06T00:21:40.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:40.081003 I | op-mon: mons running: [a b] 2022-03-06T00:21:40.295 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:40.303606 I | op-mon: Monitors in quorum: [a] 2022-03-06T00:21:40.872 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:40.880599 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-06T00:21:41.320 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:41.665 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:41.665 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 112s 2022-03-06T00:21:41.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:41.734657 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:1] 2022-03-06T00:21:41.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:41.734712 I | op-mon: waiting for mon quorum with [a b] 2022-03-06T00:21:42.360 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:42.369142 I | op-mon: mons running: [a b] 2022-03-06T00:21:42.591 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:42.599174 I | op-mon: Monitors in quorum: [a] 2022-03-06T00:21:42.592 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:42.599201 I | op-mon: mons created: 2 2022-03-06T00:21:42.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:42.960418 I | op-mon: waiting for mon quorum with [a b] 2022-03-06T00:21:42.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:42.973121 I | op-mon: mons running: [a b] 2022-03-06T00:21:43.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:43.221497 W | op-mon: failed to find initial monitor b in mon map 2022-03-06T00:21:43.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:43.221516 W | op-mon: monitor b is not in quorum list 2022-03-06T00:21:46.668 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:46.729 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:46.730 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 117s 2022-03-06T00:21:48.233 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:48.242145 I | op-mon: mons running: [a b] 2022-03-06T00:21:48.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:48.513971 W | op-mon: failed to find initial monitor b in mon map 2022-03-06T00:21:48.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:48.513991 W | op-mon: monitor b is not in quorum list 2022-03-06T00:21:51.732 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:51.801 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:51.801 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 2m2s 2022-03-06T00:21:53.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:53.533423 I | op-mon: mon b is not yet running 2022-03-06T00:21:53.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:53.533453 I | op-mon: mons running: [a] 2022-03-06T00:21:54.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408514 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-06T00:21:54.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408552 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-06T00:21:54.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408561 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:21:54.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408566 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:21:54.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408572 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:21:54.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408576 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:21:54.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408586 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:21:54.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408594 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:21:54.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408602 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:21:54.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408611 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:21:54.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408619 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:21:54.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408632 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:21:54.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408642 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:21:54.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408649 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:21:54.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408656 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-06T00:21:54.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408668 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-06T00:21:54.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408676 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-06T00:21:54.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408688 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-06T00:21:54.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408696 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-06T00:21:54.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408708 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:21:54.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408722 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:21:54.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408731 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-06T00:21:54.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408742 I | ceph-csi: Kubernetes version is 1.23 2022-03-06T00:21:54.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408770 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-06T00:21:54.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.408781 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-06T00:21:54.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.463829 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-06T00:21:54.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.474023 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:21:54.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.474067 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:21:54.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.474075 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:21:54.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.474081 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:21:54.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.474087 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:21:54.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.474093 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:21:54.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.474099 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-06T00:21:54.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.573230 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:21:54.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.573259 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:21:54.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.573268 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:21:54.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.637936 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-06T00:21:54.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.762245 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:21:54.754 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.762283 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:21:54.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.762293 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-06T00:21:54.806 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.814134 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:21:54.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.814166 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:21:54.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.814174 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:21:54.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:54.910788 I | ceph-csi: successfully started CSI CephFS driver 2022-03-06T00:21:55.065 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:55.073229 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:21:55.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:55.124047 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-06T00:21:55.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:55.124084 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:21:55.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:55.254234 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-06T00:21:56.804 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:21:56.890 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:21:56.890 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 2m7s 2022-03-06T00:21:58.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:58.656896 I | op-mon: mon b is not yet running 2022-03-06T00:21:58.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:21:58.656923 I | op-mon: mons running: [a] 2022-03-06T00:22:01.894 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:01.986 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:01.987 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 2m12s 2022-03-06T00:22:03.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:03.679596 I | op-mon: mon b is not yet running 2022-03-06T00:22:03.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:03.679619 I | op-mon: mons running: [a] 2022-03-06T00:22:06.989 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:07.106 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:07.106 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 2m18s 2022-03-06T00:22:08.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:08.696434 I | op-mon: mon b is not yet running 2022-03-06T00:22:08.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:08.696471 I | op-mon: mons running: [a] 2022-03-06T00:22:12.109 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:12.185 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:12.185 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 0/1 ContainerCreating 0 2m23s 2022-03-06T00:22:13.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:13.713561 I | op-mon: mon b is not yet running 2022-03-06T00:22:13.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:13.713586 I | op-mon: mons running: [a] 2022-03-06T00:22:17.188 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-06T00:22:17.262 INFO:teuthology.orchestra.run.smithi007.stdout:NAME READY STATUS RESTARTS AGE 2022-03-06T00:22:17.263 INFO:teuthology.orchestra.run.smithi007.stdout:rook-ceph-tools-d6d7c985c-pld5f 1/1 Running 0 2m28s 2022-03-06T00:22:17.267 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-06T00:22:17.267 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:22:17.838 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:22:17.839 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:22:17.839 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:22:17.839 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:22:17.840 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:22:17.840 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:22:17.840 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:22:17.840 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:22:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:22:17.841 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:22:17.842 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:22:17.843 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:22:17.843 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:22:17.843 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:22:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:22:19.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:19.458449 I | op-mon: mon b is not yet running 2022-03-06T00:22:19.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:19.458497 I | op-mon: mons running: [a] 2022-03-06T00:22:24.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:24.656176 I | op-mon: mons running: [a b] 2022-03-06T00:22:24.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:24.917448 I | op-mon: Monitors in quorum: [a b] 2022-03-06T00:22:24.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:24.917485 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:22:25.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:25.178330 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:22:25.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:25.178379 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:22:25.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:25.435780 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:22:25.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:25.435816 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:22:25.719 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:25.727565 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:22:25.719 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:25.727597 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:22:25.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:25.972177 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:22:25.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:25.972244 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:22:26.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:26.251467 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:22:26.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:26.251494 I | op-mon: creating mon c 2022-03-06T00:22:26.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:26.372668 I | op-mon: mon "a" endpoint is [v2:10.102.190.159:3300,v1:10.102.190.159:6789] 2022-03-06T00:22:26.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:26.553412 I | op-mon: mon "b" endpoint is [v2:10.98.23.111:3300,v1:10.98.23.111:6789] 2022-03-06T00:22:26.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:26.589278 I | op-mon: mon "c" endpoint is [v2:10.109.72.196:3300,v1:10.109.72.196:6789] 2022-03-06T00:22:26.694 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:26.704122 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789","10.109.72.196:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:1] 2022-03-06T00:22:27.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:27.255975 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:22:27.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:27.256319 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:22:27.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:27.674506 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-03-06T00:22:27.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:27.680406 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:22:27.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:27.690415 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:22:27.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:27.696004 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-06T00:22:27.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:27.710708 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-06T00:22:27.846 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:22:27.862 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:27.871698 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-06T00:22:28.342 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:22:28.343 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:22:28.343 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:22:28.344 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:22:28.344 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:22:28.344 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:22:28.345 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:22:28.345 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:22:28.345 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:22:28.345 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:22:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:22:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:22:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:22:28.347 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:22:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:22:28.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:28.656590 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789","10.109.72.196:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-06T00:22:28.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:28.656618 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:22:29.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:29.423829 I | op-mon: mon c is not yet running 2022-03-06T00:22:29.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:29.423870 I | op-mon: mons running: [a b] 2022-03-06T00:22:29.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:29.676805 I | op-mon: Monitors in quorum: [a b] 2022-03-06T00:22:29.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:29.676835 I | op-mon: mons created: 3 2022-03-06T00:22:29.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:29.952073 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:22:29.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:29.984756 I | op-mon: mon c is not yet running 2022-03-06T00:22:29.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:29.984787 I | op-mon: mons running: [a b] 2022-03-06T00:22:35.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:35.017130 I | op-mon: mon c is not yet running 2022-03-06T00:22:35.008 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:35.017167 I | op-mon: mons running: [a b] 2022-03-06T00:22:38.351 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:22:38.894 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:22:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:22:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:22:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:22:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:22:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:22:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:22:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:22:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:22:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:22:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:22:38.897 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:22:38.897 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:22:38.898 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:22:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:22:40.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:40.038047 I | op-mon: mon c is not yet running 2022-03-06T00:22:40.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:40.038081 I | op-mon: mons running: [a b] 2022-03-06T00:22:45.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:45.058393 I | op-mon: mon c is not yet running 2022-03-06T00:22:45.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:45.058426 I | op-mon: mons running: [a b] 2022-03-06T00:22:48.902 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:22:49.426 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:22:49.426 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:22:49.427 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:22:49.427 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:22:49.427 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:22:49.427 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:22:49.427 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:22:49.428 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:22:49.428 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:22:49.428 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:22:49.428 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:22:49.429 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:22:49.429 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:22:49.429 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:22:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:22:50.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:50.306781 I | op-mon: mon c is not yet running 2022-03-06T00:22:50.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:50.306818 I | op-mon: mons running: [a b] 2022-03-06T00:22:55.882 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:55.892436 I | op-mon: mon c is not yet running 2022-03-06T00:22:55.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:22:55.892462 I | op-mon: mons running: [a b] 2022-03-06T00:22:59.432 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:23:00.145 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:23:00.146 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:23:00.146 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:23:00.146 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:23:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:23:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:23:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:23:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:23:00.147 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:23:00.148 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:23:00.148 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:23:00.148 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:23:00.149 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:23:00.151 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:23:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:23:01.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:01.202860 I | op-mon: mon c is not yet running 2022-03-06T00:23:01.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:01.202885 I | op-mon: mons running: [a b] 2022-03-06T00:23:07.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:07.868469 I | op-mon: mon c is not yet running 2022-03-06T00:23:07.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:07.868497 I | op-mon: mons running: [a b] 2022-03-06T00:23:10.154 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:23:11.209 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:23:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:23:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:23:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:23:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:23:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:23:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:23:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:23:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:23:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:23:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:23:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:23:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:23:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:23:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:23:14.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:14.419276 I | op-mon: mon c is not yet running 2022-03-06T00:23:14.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:14.419307 I | op-mon: mons running: [a b] 2022-03-06T00:23:19.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:19.934121 I | op-mon: mon c is not yet running 2022-03-06T00:23:19.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:19.934164 I | op-mon: mons running: [a b] 2022-03-06T00:23:21.218 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:23:22.735 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:23:22.736 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:23:22.737 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:23:22.737 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:23:22.738 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:23:22.738 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:23:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:23:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:23:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:23:22.740 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:23:22.740 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:23:22.741 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:23:22.741 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:23:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:23:22.744 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:23:27.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:27.037183 I | op-mon: mon c is not yet running 2022-03-06T00:23:27.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:27.037231 I | op-mon: mons running: [a b] 2022-03-06T00:23:32.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:32.185774 I | op-mon: mon c is not yet running 2022-03-06T00:23:32.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:32.185799 I | op-mon: mons running: [a b] 2022-03-06T00:23:32.745 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:23:33.335 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:23:33.336 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:23:33.336 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:23:33.336 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:23:33.336 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:23:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:23:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:23:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:23:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:23:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:23:33.338 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:23:33.338 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:23:33.338 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:23:33.340 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:23:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:23:37.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:37.208047 I | op-mon: mon c is not yet running 2022-03-06T00:23:37.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:37.208088 I | op-mon: mons running: [a b] 2022-03-06T00:23:42.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:42.231705 I | op-mon: mons running: [a b c] 2022-03-06T00:23:42.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:42.687850 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:23:42.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:42.687889 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:23:42.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:42.947266 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:23:42.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:42.947299 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:23:43.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:43.192579 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:23:43.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:43.192616 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:23:43.345 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:23:43.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:43.423061 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:23:43.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:43.423090 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:23:43.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:43.755868 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:23:43.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:43.755903 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:23:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:23:43.834 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:23:43.834 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:23:43.834 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:23:43.834 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:23:43.835 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:23:43.835 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:23:43.835 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:23:43.835 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:23:43.836 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:23:43.836 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:23:43.836 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:23:43.836 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:23:43.837 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:23:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:23:43.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:43.992642 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:23:43.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:43.992690 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-06T00:23:44.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:44.341508 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-06T00:23:44.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:44.700421 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-06T00:23:45.084 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:45.094946 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-06T00:23:45.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:45.498605 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-06T00:23:45.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:45.498637 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-06T00:23:45.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:45.896088 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-06T00:23:46.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:46.120968 I | cephclient: successfully enabled msgr2 protocol 2022-03-06T00:23:46.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:46.121016 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-06T00:23:46.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:46.121028 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-06T00:23:46.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:46.469804 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-06T00:23:46.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:46.497783 I | op-mgr: start running mgr 2022-03-06T00:23:46.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:46.497829 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-06T00:23:47.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:47.251848 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:23:47.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:23:47.333248 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:23:53.839 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:23:54.344 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:23:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:no valid command found; 10 closest matches: 2022-03-06T00:23:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:pg stat 2022-03-06T00:23:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:pg getmap 2022-03-06T00:23:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-06T00:23:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-06T00:23:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_pools_json 2022-03-06T00:23:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-pool [...] 2022-03-06T00:23:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-primary [] [...] 2022-03-06T00:23:54.347 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls-by-osd [] [...] 2022-03-06T00:23:54.347 INFO:teuthology.orchestra.run.smithi007.stderr:pg ls [] [...] 2022-03-06T00:23:54.347 INFO:teuthology.orchestra.run.smithi007.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-06T00:23:54.347 INFO:teuthology.orchestra.run.smithi007.stderr:Error EINVAL: invalid command 2022-03-06T00:23:54.349 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 22 2022-03-06T00:23:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-06T00:24:04.354 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:24:05.028 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:24:05.029 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-06T00:24:05.035 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 2 2022-03-06T00:24:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-06T00:24:07.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:07.959014 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-06T00:24:07.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:07.962778 I | op-mgr: setting services to point to mgr "a" 2022-03-06T00:24:07.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.007611 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-06T00:24:07.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.007714 I | op-mgr: successful modules: balancer 2022-03-06T00:24:08.008 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.019245 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-06T00:24:08.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.019279 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-06T00:24:08.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.019293 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-06T00:24:08.026 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.037222 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-06T00:24:08.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.040548 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-06T00:24:08.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.040577 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-06T00:24:08.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.040584 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-06T00:24:08.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.738825 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-06T00:24:08.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.738854 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:24:08.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.745091 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-06T00:24:08.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.745130 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-06T00:24:08.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:08.745146 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-06T00:24:09.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:09.454221 I | op-mgr: successful modules: prometheus 2022-03-06T00:24:09.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:09.454789 I | op-mgr: successful modules: dashboard 2022-03-06T00:24:09.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:09.500458 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-06T00:24:15.039 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:24:17.814 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:24:17.817 INFO:teuthology.orchestra.run.smithi007.stderr:Error ENOENT: No orchestrator configured (try `ceph orch set backend`) 2022-03-06T00:24:17.822 INFO:teuthology.orchestra.run.smithi007.stderr:command terminated with exit code 2 2022-03-06T00:24:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-03-06T00:24:17.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:17.931117 I | op-mgr: successful modules: orchestrator modules 2022-03-06T00:24:27.825 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status -f json 2022-03-06T00:24:28.498 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:24:29.265 INFO:teuthology.orchestra.run.smithi007.stdout:{"available": true, "backend": "rook"} 2022-03-06T00:24:29.268 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-06T00:24:29.268 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-06T00:24:30.140 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch apply osd --all-available-devices 2022-03-06T00:24:31.018 INFO:teuthology.orchestra.run.smithi007.stdout:['Success'] 2022-03-06T00:24:31.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.031707 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-06T00:24:31.021 INFO:tasks.rook.operator.smithi007.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", AllowUnsupported: true}, 2022-03-06T00:24:31.021 INFO:tasks.rook.operator.smithi007.stdout: Storage: v1.StorageScopeSpec{ 2022-03-06T00:24:31.021 INFO:tasks.rook.operator.smithi007.stdout: ... // 3 identical fields 2022-03-06T00:24:31.022 INFO:tasks.rook.operator.smithi007.stdout: Config: nil, 2022-03-06T00:24:31.023 INFO:tasks.rook.operator.smithi007.stdout: Selection: {}, 2022-03-06T00:24:31.024 INFO:tasks.rook.operator.smithi007.stdout:- StorageClassDeviceSets: nil, 2022-03-06T00:24:31.024 INFO:tasks.rook.operator.smithi007.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-06T00:24:31.024 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.024 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi007-lv-1", 2022-03-06T00:24:31.024 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.025 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.025 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.025 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.025 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi007-lv-2", 2022-03-06T00:24:31.026 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.026 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.031 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.031 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.032 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi007-lv-3", 2022-03-06T00:24:31.032 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.032 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.032 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.033 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.033 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi007-lv-4", 2022-03-06T00:24:31.033 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.033 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.033 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.034 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.034 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi017-lv-1", 2022-03-06T00:24:31.034 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.034 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.035 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.035 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.035 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi017-lv-2", 2022-03-06T00:24:31.035 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.035 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.036 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.036 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.036 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi017-lv-3", 2022-03-06T00:24:31.036 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.037 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.037 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.037 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.037 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi017-lv-4", 2022-03-06T00:24:31.037 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.038 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.038 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.038 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.038 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi066-lv-1", 2022-03-06T00:24:31.038 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.039 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.039 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.039 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.039 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi066-lv-2", 2022-03-06T00:24:31.040 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.040 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.040 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.040 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.040 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi066-lv-3", 2022-03-06T00:24:31.041 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.041 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.041 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.041 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:24:31.042 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi066-lv-4", 2022-03-06T00:24:31.042 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:24:31.042 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-06T00:24:31.042 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.042 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:24:31.043 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-06T00:24:31.043 INFO:tasks.rook.operator.smithi007.stdout: Annotations: nil, 2022-03-06T00:24:31.043 INFO:tasks.rook.operator.smithi007.stdout: Labels: nil, 2022-03-06T00:24:31.043 INFO:tasks.rook.operator.smithi007.stdout: ... // 20 identical fields 2022-03-06T00:24:31.044 INFO:tasks.rook.operator.smithi007.stdout: } 2022-03-06T00:24:31.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.031843 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-06T00:24:31.044 INFO:tasks.rook.operator.smithi007.stdout:I0306 00:24:31.032388 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-06T00:24:31.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.032513 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-06T00:24:31.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.032573 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-06T00:24:31.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.032592 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-06T00:24:31.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.032639 I | operator: successfully started the controller-runtime manager 2022-03-06T00:24:31.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.046917 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:24:31.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.046946 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:24:31.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.046995 I | operator: setting up schemes 2022-03-06T00:24:31.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.050498 I | operator: setting up the controller-runtime manager 2022-03-06T00:24:31.048 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-06T00:24:31.048 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:24:31.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.353917 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:24:31.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356443 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:24:31.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356483 I | ceph-cluster-controller: successfully started 2022-03-06T00:24:31.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356559 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:24:31.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356590 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:24:31.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356628 I | ceph-block-pool-controller: successfully started 2022-03-06T00:24:31.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356668 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:24:31.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356705 I | ceph-object-realm-controller: successfully started 2022-03-06T00:24:31.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356730 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:24:31.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356754 I | ceph-object-zone-controller: successfully started 2022-03-06T00:24:31.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356907 I | ceph-object-controller: successfully started 2022-03-06T00:24:31.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.356964 I | ceph-file-controller: successfully started 2022-03-06T00:24:31.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357008 I | ceph-nfs-controller: successfully started 2022-03-06T00:24:31.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357052 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:24:31.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357091 I | ceph-client-controller: successfully started 2022-03-06T00:24:31.352 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357115 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:24:31.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357145 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:24:31.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357168 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:24:31.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357185 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:24:31.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357202 I | ceph-bucket-topic: successfully started 2022-03-06T00:24:31.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357214 I | ceph-bucket-notification: successfully started 2022-03-06T00:24:31.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357233 I | ceph-bucket-notification: successfully started 2022-03-06T00:24:31.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.357250 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:24:31.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.358543 I | operator: starting the controller-runtime manager 2022-03-06T00:24:31.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.463494 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:24:31.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.464085 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:24:31.458 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.464146 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:24:31.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.464163 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:24:31.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.468703 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:24:31.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.471514 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:24:31.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.471536 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:24:31.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.471558 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-06T00:24:31.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.471568 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:24:31.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.472749 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-06T00:24:31.464 INFO:tasks.rook.operator.smithi007.stdout:I0306 00:24:31.472816 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:24:31.500 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.511651 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-06T00:24:31.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.515642 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:24:31.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.541938 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-06T00:24:31.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.541972 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-06T00:24:31.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.541986 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-06T00:24:31.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.541998 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-06T00:24:31.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542015 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-06T00:24:31.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542057 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-06T00:24:31.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542079 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-06T00:24:31.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542098 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-06T00:24:31.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542118 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-06T00:24:31.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542136 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-06T00:24:31.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542153 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-06T00:24:31.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542172 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-06T00:24:31.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542190 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-06T00:24:31.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542202 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-06T00:24:31.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542215 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-06T00:24:31.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542239 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-06T00:24:31.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542338 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:24:31.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:31.542358 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:24:31.611 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:24:31.631 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:24:31.632 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:24:37.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.898954 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-06T00:24:37.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.898996 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-06T00:24:37.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899008 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:24:37.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899015 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:24:37.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899022 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:24:37.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899029 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:24:37.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899036 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:24:37.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899042 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:24:37.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899049 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:24:37.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899055 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:24:37.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899062 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:24:37.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899069 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:24:37.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899076 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:24:37.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899083 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:24:37.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899090 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-06T00:24:37.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899097 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-06T00:24:37.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899104 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-06T00:24:37.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899111 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-06T00:24:37.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899118 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-06T00:24:37.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899126 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:24:37.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899133 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:24:37.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899141 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-06T00:24:37.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899148 I | ceph-csi: Kubernetes version is 1.23 2022-03-06T00:24:37.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899158 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-06T00:24:37.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.899165 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-06T00:24:37.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.930537 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-06T00:24:37.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.942108 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:24:37.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.942132 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:24:37.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.942136 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:24:37.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.942139 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:24:37.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.942142 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:24:37.934 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.942145 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:24:37.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:37.942148 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-06T00:24:38.054 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:38.065511 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:24:38.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:38.065553 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:24:38.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:38.065565 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:24:38.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:38.108449 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-06T00:24:38.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:38.664068 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:24:38.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:38.664102 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:24:38.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:38.664112 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-06T00:24:39.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.138964 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:24:39.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.138989 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:24:39.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.138998 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:24:39.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.223513 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:24:39.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.223544 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:24:39.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.223555 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:24:39.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.297640 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:24:39.446 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.457059 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:24:39.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.457397 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:24:39.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.480145 I | ceph-csi: successfully started CSI CephFS driver 2022-03-06T00:24:39.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.715219 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:24:39.719 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.730407 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-06T00:24:39.719 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.730435 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:24:39.761 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.772247 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-06T00:24:39.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:39.888936 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:24:40.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:40.019779 I | op-mon: start running mons 2022-03-06T00:24:40.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:40.102098 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:24:40.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:40.167535 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789","10.109.72.196:6789"]}] data:c=10.109.72.196:6789,a=10.102.190.159:6789,b=10.98.23.111:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-06T00:24:40.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:40.617343 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:24:40.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:40.617651 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:24:41.633 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:24:42.146 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:24:42.166 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:24:42.167 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:24:42.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:42.186890 I | op-mon: targeting the mon count 3 2022-03-06T00:24:42.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:42.191304 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:24:42.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:42.535556 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:24:42.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:42.535586 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:24:42.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:42.962471 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:24:42.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:42.962510 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:24:43.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:43.316261 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:24:43.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:43.316315 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:24:43.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:43.666557 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:24:43.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:43.666590 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:24:44.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:44.053324 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:24:44.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:44.053355 I | op-mon: checking for basic quorum with existing mons 2022-03-06T00:24:44.245 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:44.256293 I | op-mon: mon "a" endpoint is [v2:10.102.190.159:3300,v1:10.102.190.159:6789] 2022-03-06T00:24:44.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:44.436258 I | op-mon: mon "b" endpoint is [v2:10.98.23.111:3300,v1:10.98.23.111:6789] 2022-03-06T00:24:44.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:44.620512 I | op-mon: mon "c" endpoint is [v2:10.109.72.196:3300,v1:10.109.72.196:6789] 2022-03-06T00:24:45.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:45.188009 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.23.111:6789","10.109.72.196:6789","10.102.190.159:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-06T00:24:45.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:45.786599 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:24:45.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:45.786879 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:24:46.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:46.214384 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:24:46.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:46.223098 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:24:46.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:46.223122 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:24:46.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:46.794312 I | op-mon: mons running: [a b c] 2022-03-06T00:24:47.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.235567 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:24:47.229 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.241229 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-06T00:24:47.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.250953 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-06T00:24:47.240 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.250977 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:24:47.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.391108 I | op-mon: mons running: [a b c] 2022-03-06T00:24:47.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.830989 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:24:47.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.840422 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-06T00:24:47.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.850717 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-06T00:24:47.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.850743 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:24:47.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:47.991610 I | op-mon: mons running: [a b c] 2022-03-06T00:24:48.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:48.414388 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:24:48.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:48.414420 I | op-mon: mons created: 3 2022-03-06T00:24:48.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:48.813081 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:24:48.828 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:48.839333 I | op-mon: mons running: [a b c] 2022-03-06T00:24:49.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:49.253168 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:24:49.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:49.253212 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-06T00:24:49.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:49.732723 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-06T00:24:50.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:50.177429 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-06T00:24:50.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:50.590976 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-06T00:24:51.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:51.131171 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-06T00:24:51.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:51.131204 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-06T00:24:51.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:51.566152 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-06T00:24:51.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:51.986783 I | cephclient: successfully enabled msgr2 protocol 2022-03-06T00:24:51.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:51.986821 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-06T00:24:51.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:51.986830 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-06T00:24:52.168 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:24:52.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:52.438180 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-06T00:24:52.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:52.485166 I | op-mgr: start running mgr 2022-03-06T00:24:52.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:52.485201 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-06T00:24:52.686 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:24:52.705 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:24:52.706 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:24:52.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:52.963850 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-06T00:24:52.964 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:52.975388 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-06T00:24:52.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.003747 I | op-mgr: setting services to point to mgr "a" 2022-03-06T00:24:53.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.131057 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-06T00:24:53.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.131178 I | op-mgr: successful modules: balancer 2022-03-06T00:24:53.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.149948 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-06T00:24:53.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.149996 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-06T00:24:53.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.156954 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-06T00:24:53.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.310414 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-1" 2022-03-06T00:24:53.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.514873 I | op-osd: successfully provisioned PVC "smithi007-lv-1-data-07jksx" 2022-03-06T00:24:53.504 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.514906 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-2" 2022-03-06T00:24:53.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.744045 I | op-osd: successfully provisioned PVC "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:24:53.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.744100 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-3" 2022-03-06T00:24:53.901 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.912574 I | op-osd: successfully provisioned PVC "smithi007-lv-3-data-059rmw" 2022-03-06T00:24:53.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:53.912612 I | op-osd: creating 1 new PVCs for device set "smithi007-lv-4" 2022-03-06T00:24:54.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.143557 I | op-osd: successfully provisioned PVC "smithi007-lv-4-data-0nknbb" 2022-03-06T00:24:54.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.143601 I | op-osd: creating 1 new PVCs for device set "smithi017-lv-1" 2022-03-06T00:24:54.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.312790 I | op-osd: successfully provisioned PVC "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:24:54.302 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.312829 I | op-osd: creating 1 new PVCs for device set "smithi017-lv-2" 2022-03-06T00:24:54.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.499665 I | op-mgr: successful modules: dashboard 2022-03-06T00:24:54.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.526862 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-06T00:24:54.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.569180 I | op-osd: successfully provisioned PVC "smithi017-lv-2-data-04dp76" 2022-03-06T00:24:54.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.569218 I | op-osd: creating 1 new PVCs for device set "smithi017-lv-3" 2022-03-06T00:24:54.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.597979 I | op-mgr: successful modules: prometheus 2022-03-06T00:24:54.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.717688 I | op-osd: successfully provisioned PVC "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:24:54.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.717725 I | op-osd: creating 1 new PVCs for device set "smithi017-lv-4" 2022-03-06T00:24:54.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.910092 I | op-osd: successfully provisioned PVC "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:24:54.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:54.910110 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-1" 2022-03-06T00:24:55.100 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:55.111915 I | op-osd: successfully provisioned PVC "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:24:55.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:55.111957 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-2" 2022-03-06T00:24:55.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:55.312252 I | op-osd: successfully provisioned PVC "smithi066-lv-2-data-04bnc5" 2022-03-06T00:24:55.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:55.312285 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-3" 2022-03-06T00:24:55.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:55.512216 I | op-osd: successfully provisioned PVC "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:24:55.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:55.512258 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-4" 2022-03-06T00:24:55.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:55.711136 I | op-osd: successfully provisioned PVC "smithi066-lv-4-data-0bh4ll" 2022-03-06T00:24:55.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:55.737537 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-07jksx" 2022-03-06T00:24:56.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:56.103544 I | op-mgr: successful modules: orchestrator modules 2022-03-06T00:24:56.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:56.144309 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-1-data-07jksx" 2022-03-06T00:24:56.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:56.144338 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:24:56.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:56.557633 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:24:56.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:56.557668 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-059rmw" 2022-03-06T00:24:57.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:57.267282 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-3-data-059rmw" 2022-03-06T00:24:57.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:57.267318 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0nknbb" 2022-03-06T00:24:57.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:57.803425 I | op-osd: started OSD provisioning job for PVC "smithi007-lv-4-data-0nknbb" 2022-03-06T00:24:57.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:57.803472 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:24:58.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:58.185840 I | op-osd: started OSD provisioning job for PVC "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:24:58.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:58.185874 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-2-data-04dp76" 2022-03-06T00:24:59.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:59.202160 I | op-osd: started OSD provisioning job for PVC "smithi017-lv-2-data-04dp76" 2022-03-06T00:24:59.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:24:59.202194 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:25:00.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:00.056637 I | op-osd: started OSD provisioning job for PVC "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:25:00.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:00.056690 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:25:00.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:00.997670 I | op-osd: started OSD provisioning job for PVC "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:25:00.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:00.997715 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:25:01.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:01.399951 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:25:01.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:01.399991 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-04bnc5" 2022-03-06T00:25:01.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:01.601414 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-2-data-04bnc5" 2022-03-06T00:25:01.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:01.601447 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:25:01.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:01.818992 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:25:01.808 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:01.819015 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-0bh4ll" 2022-03-06T00:25:02.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.022881 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-4-data-0bh4ll" 2022-03-06T00:25:02.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.022907 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-06T00:25:02.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.022911 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-06T00:25:02.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328283 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-07jksx is "starting" 2022-03-06T00:25:02.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328334 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-0xh6k5 is "starting" 2022-03-06T00:25:02.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328356 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-059rmw is "starting" 2022-03-06T00:25:02.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328375 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0nknbb is "starting" 2022-03-06T00:25:02.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328392 I | op-osd: OSD orchestration status for PVC smithi017-lv-1-data-0dfdbl is "starting" 2022-03-06T00:25:02.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328425 I | op-osd: OSD orchestration status for PVC smithi017-lv-2-data-04dp76 is "starting" 2022-03-06T00:25:02.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328441 I | op-osd: OSD orchestration status for PVC smithi017-lv-3-data-0vkmbc is "starting" 2022-03-06T00:25:02.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328465 I | op-osd: OSD orchestration status for PVC smithi017-lv-4-data-0ncm7h is "starting" 2022-03-06T00:25:02.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328485 I | op-osd: OSD orchestration status for PVC smithi066-lv-1-data-0r6fcr is "starting" 2022-03-06T00:25:02.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328503 I | op-osd: OSD orchestration status for PVC smithi066-lv-2-data-04bnc5 is "starting" 2022-03-06T00:25:02.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328523 I | op-osd: OSD orchestration status for PVC smithi066-lv-3-data-0mh6gp is "starting" 2022-03-06T00:25:02.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:02.328541 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bh4ll is "starting" 2022-03-06T00:25:02.709 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:25:03.610 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:25:03.635 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:25:03.635 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:25:12.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:12.678257 I | op-osd: OSD orchestration status for node smithi066-lv-4-data-0bh4ll is "orchestrating" 2022-03-06T00:25:13.366 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:13.374140 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bh4ll is "orchestrating" 2022-03-06T00:25:13.637 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:25:14.922 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:25:14.942 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-06T00:25:14.943 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:25:15.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:15.738811 I | op-osd: OSD orchestration status for node smithi007-lv-1-data-07jksx is "orchestrating" 2022-03-06T00:25:15.763 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:15.766636 I | op-osd: OSD orchestration status for node smithi017-lv-2-data-04dp76 is "orchestrating" 2022-03-06T00:25:16.460 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:16.401947 I | op-osd: OSD orchestration status for PVC smithi017-lv-2-data-04dp76 is "orchestrating" 2022-03-06T00:25:16.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:16.402105 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-07jksx is "orchestrating" 2022-03-06T00:25:16.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:16.477532 I | op-osd: OSD orchestration status for node smithi007-lv-3-data-059rmw is "orchestrating" 2022-03-06T00:25:16.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:16.878969 I | op-osd: OSD orchestration status for node smithi007-lv-4-data-0nknbb is "orchestrating" 2022-03-06T00:25:16.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:16.940704 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-059rmw is "orchestrating" 2022-03-06T00:25:17.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:17.444721 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0nknbb is "orchestrating" 2022-03-06T00:25:18.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:18.379192 I | op-osd: OSD orchestration status for node smithi017-lv-4-data-0ncm7h is "orchestrating" 2022-03-06T00:25:18.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:18.936253 I | op-osd: OSD orchestration status for PVC smithi017-lv-4-data-0ncm7h is "orchestrating" 2022-03-06T00:25:18.946 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:18.952700 I | op-osd: OSD orchestration status for node smithi017-lv-3-data-0vkmbc is "orchestrating" 2022-03-06T00:25:19.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:19.426573 I | op-osd: OSD orchestration status for PVC smithi017-lv-3-data-0vkmbc is "orchestrating" 2022-03-06T00:25:22.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:22.130907 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bh4ll is "completed" 2022-03-06T00:25:22.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:22.130944 I | op-osd: creating OSD 0 on PVC "smithi066-lv-4-data-0bh4ll" 2022-03-06T00:25:22.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:22.130963 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-0bh4ll" 2022-03-06T00:25:23.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:23.072448 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:25:23.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:23.523301 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:25:24.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:24.193905 I | op-osd: OSD orchestration status for node smithi017-lv-1-data-0dfdbl is "orchestrating" 2022-03-06T00:25:24.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:24.450698 I | op-osd: OSD orchestration status for node smithi066-lv-1-data-0r6fcr is "orchestrating" 2022-03-06T00:25:24.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:24.841569 I | op-osd: OSD orchestration status for PVC smithi017-lv-1-data-0dfdbl is "orchestrating" 2022-03-06T00:25:24.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:24.841709 I | op-osd: OSD orchestration status for PVC smithi007-lv-1-data-07jksx is "completed" 2022-03-06T00:25:24.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:24.841729 I | op-osd: creating OSD 1 on PVC "smithi007-lv-1-data-07jksx" 2022-03-06T00:25:24.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:24.841747 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-07jksx" 2022-03-06T00:25:24.944 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:25:25.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:25.582640 I | op-osd: OSD orchestration status for PVC smithi066-lv-1-data-0r6fcr is "orchestrating" 2022-03-06T00:25:25.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:25.747049 I | op-osd: OSD orchestration status for node smithi007-lv-2-data-0xh6k5 is "orchestrating" 2022-03-06T00:25:26.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:25:26.245 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":9,"num_osds":7,"num_up_osds":0,"osd_up_since":0,"num_in_osds":7,"osd_in_since":1646526323,"num_remapped_pgs":0} 2022-03-06T00:25:26.245 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:25:26.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:26.442500 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-0xh6k5 is "orchestrating" 2022-03-06T00:25:27.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:27.585721 I | op-osd: OSD orchestration status for node smithi066-lv-2-data-04bnc5 is "orchestrating" 2022-03-06T00:25:27.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:28.003792 I | op-osd: OSD orchestration status for PVC smithi007-lv-4-data-0nknbb is "completed" 2022-03-06T00:25:28.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:28.003835 I | op-osd: creating OSD 3 on PVC "smithi007-lv-4-data-0nknbb" 2022-03-06T00:25:28.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:28.003850 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0nknbb" 2022-03-06T00:25:29.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:29.685977 I | op-osd: OSD orchestration status for PVC smithi017-lv-2-data-04dp76 is "completed" 2022-03-06T00:25:29.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:29.686018 I | op-osd: creating OSD 4 on PVC "smithi017-lv-2-data-04dp76" 2022-03-06T00:25:29.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:29.686089 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-2-data-04dp76" 2022-03-06T00:25:30.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:30.572974 I | op-osd: OSD orchestration status for PVC smithi007-lv-3-data-059rmw is "completed" 2022-03-06T00:25:30.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:30.572996 I | op-osd: creating OSD 2 on PVC "smithi007-lv-3-data-059rmw" 2022-03-06T00:25:30.571 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:30.573004 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-059rmw" 2022-03-06T00:25:31.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:31.090774 I | op-osd: OSD orchestration status for PVC smithi066-lv-2-data-04bnc5 is "orchestrating" 2022-03-06T00:25:31.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:31.091578 I | op-osd: OSD orchestration status for PVC smithi017-lv-4-data-0ncm7h is "completed" 2022-03-06T00:25:31.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:31.091602 I | op-osd: creating OSD 5 on PVC "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:25:31.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:31.091618 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:25:34.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:34.251264 I | op-osd: OSD orchestration status for PVC smithi017-lv-3-data-0vkmbc is "completed" 2022-03-06T00:25:34.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:34.251318 I | op-osd: creating OSD 6 on PVC "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:25:34.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:34.251344 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:25:35.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:35.359388 I | op-osd: OSD orchestration status for node smithi066-lv-3-data-0mh6gp is "orchestrating" 2022-03-06T00:25:35.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:35.360436 I | op-osd: OSD orchestration status for PVC smithi066-lv-3-data-0mh6gp is "orchestrating" 2022-03-06T00:25:35.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:35.360678 I | op-osd: OSD orchestration status for PVC smithi066-lv-1-data-0r6fcr is "completed" 2022-03-06T00:25:35.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:35.360703 I | op-osd: creating OSD 7 on PVC "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:25:35.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:35.360723 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:25:36.247 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:25:37.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:37.413912 I | op-osd: OSD orchestration status for PVC smithi017-lv-1-data-0dfdbl is "completed" 2022-03-06T00:25:37.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:37.413951 I | op-osd: creating OSD 8 on PVC "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:25:37.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:37.413966 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:25:37.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:37.979511 I | op-osd: OSD orchestration status for PVC smithi007-lv-2-data-0xh6k5 is "completed" 2022-03-06T00:25:37.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:37.979531 I | op-osd: creating OSD 9 on PVC "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:25:37.978 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:37.979538 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:25:38.252 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:25:38.274 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646526332,"num_remapped_pgs":0} 2022-03-06T00:25:38.274 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:25:39.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:39.336218 I | op-osd: OSD orchestration status for PVC smithi066-lv-2-data-04bnc5 is "completed" 2022-03-06T00:25:39.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:39.336288 I | op-osd: creating OSD 10 on PVC "smithi066-lv-2-data-04bnc5" 2022-03-06T00:25:39.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:39.336304 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-04bnc5" 2022-03-06T00:25:41.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:41.079026 I | op-osd: OSD orchestration status for PVC smithi066-lv-3-data-0mh6gp is "completed" 2022-03-06T00:25:41.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:41.079068 I | op-osd: creating OSD 11 on PVC "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:25:41.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:41.079088 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:25:42.421 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:42.422635 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:25:43.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:43.360778 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:25:44.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:44.167592 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-06T00:25:44.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:25:44.167627 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:25:48.276 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:25:50.490 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:25:50.511 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646526332,"num_remapped_pgs":0} 2022-03-06T00:25:50.511 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:26:00.512 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:26:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:02.484 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646526332,"num_remapped_pgs":0} 2022-03-06T00:26:02.485 INFO:tasks.rook: have 0/12 OSDs 2022-03-06T00:26:12.486 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:26:13.008 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:13.037 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"num_osds":12,"num_up_osds":9,"osd_up_since":1646526372,"num_in_osds":12,"osd_in_since":1646526332,"num_remapped_pgs":0} 2022-03-06T00:26:13.037 INFO:tasks.rook: have 9/12 OSDs 2022-03-06T00:26:23.038 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd stat -f json 2022-03-06T00:26:23.547 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:23.569 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"num_osds":12,"num_up_osds":12,"osd_up_since":1646526375,"num_in_osds":12,"osd_in_since":1646526332,"num_remapped_pgs":0} 2022-03-06T00:26:23.569 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-06T00:26:23.570 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- cat /etc/ceph/ceph.conf 2022-03-06T00:26:23.773 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2022-03-06T00:26:23.774 INFO:teuthology.orchestra.run.smithi007.stdout:mon_host = 10.102.190.159:6789,10.98.23.111:6789,10.109.72.196:6789 2022-03-06T00:26:23.774 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:23.774 INFO:teuthology.orchestra.run.smithi007.stdout:[client.admin] 2022-03-06T00:26:23.775 INFO:teuthology.orchestra.run.smithi007.stdout:keyring = /etc/ceph/keyring 2022-03-06T00:26:23.778 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- cat /etc/ceph/keyring 2022-03-06T00:26:23.960 INFO:teuthology.orchestra.run.smithi007.stdout:[client.admin] 2022-03-06T00:26:23.961 INFO:teuthology.orchestra.run.smithi007.stdout:key = AQBz/iNiztrcJxAAXL8TjTindPL9gk4sl2wr7Q== 2022-03-06T00:26:23.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2022-03-06T00:26:23.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph 2022-03-06T00:26:23.998 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph 2022-03-06T00:26:24.016 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:26:24.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2022-03-06T00:26:24.032 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:26:24.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/keyring 2022-03-06T00:26:24.047 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:26:24.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.conf 2022-03-06T00:26:24.064 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:26:24.065 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/keyring 2022-03-06T00:26:24.077 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:26:24.078 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.conf 2022-03-06T00:26:24.095 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:26:24.095 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/keyring 2022-03-06T00:26:24.107 INFO:tasks.rook:Setting up client nodes... 2022-03-06T00:26:24.108 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-06T00:26:24.993 INFO:teuthology.orchestra.run.smithi007.stdout:[client.a] 2022-03-06T00:26:24.993 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQCw/yNizWJoKRAAtCrFSXGsZXE4z+OTXoRHEg== 2022-03-06T00:26:25.019 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2022-03-06T00:26:25.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-06T00:26:25.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-06T00:26:25.047 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-06T00:26:25.777 INFO:teuthology.orchestra.run.smithi007.stdout:[client.b] 2022-03-06T00:26:25.778 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQCx/yNi2w+yJRAAaKvMUmWkbHGpzSejQ0vwng== 2022-03-06T00:26:25.803 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2022-03-06T00:26:25.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-06T00:26:25.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-06T00:26:25.825 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-06T00:26:26.654 INFO:teuthology.orchestra.run.smithi007.stdout:[client.c] 2022-03-06T00:26:26.655 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQCy/yNixiI+IBAA1/y0K2vdKAfoGRZSU2Gd8Q== 2022-03-06T00:26:26.686 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2022-03-06T00:26:26.686 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-06T00:26:26.686 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-06T00:26:26.711 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd dump --format=json 2022-03-06T00:26:27.322 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:27.323 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"b043b050-8999-48ec-a9a6-f3a813dbfc59","created":"2022-03-06T00:21:21.513204+0000","modified":"2022-03-06T00:26:18.089908+0000","last_up_change":"2022-03-06T00:26:15.925048+0000","last_in_change":"2022-03-06T00:25:32.853185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-06T00:26:11.293986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7e27b24f-261a-4450-9640-f4ba4019814d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6800","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6801","nonce":1924127139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6802","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6803","nonce":1924127139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6806","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6807","nonce":1924127139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6804","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6805","nonce":1924127139}]},"public_addr":"10.248.50.78:6801/1924127139","cluster_addr":"10.248.50.78:6803/1924127139","heartbeat_back_addr":"10.248.50.78:6807/1924127139","heartbeat_front_addr":"10.248.50.78:6805/1924127139","state":["exists","up"]},{"osd":1,"uuid":"7ba5e3e3-9fd0-4d84-bd54-acc99a222f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6800","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6801","nonce":1302588422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6802","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6803","nonce":1302588422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6806","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6807","nonce":1302588422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6804","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6805","nonce":1302588422}]},"public_addr":"10.248.54.138:6801/1302588422","cluster_addr":"10.248.54.138:6803/1302588422","heartbeat_back_addr":"10.248.54.138:6807/1302588422","heartbeat_front_addr":"10.248.54.138:6805/1302588422","state":["exists","up"]},{"osd":2,"uuid":"88598048-6d49-40db-8e1c-71dcd43867ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6800","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6801","nonce":924982503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6802","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6803","nonce":924982503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6806","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6807","nonce":924982503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6804","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6805","nonce":924982503}]},"public_addr":"10.248.54.140:6801/924982503","cluster_addr":"10.248.54.140:6803/924982503","heartbeat_back_addr":"10.248.54.140:6807/924982503","heartbeat_front_addr":"10.248.54.140:6805/924982503","state":["exists","up"]},{"osd":3,"uuid":"fd7c4212-ca13-4464-8fa2-de0e42305500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6800","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6801","nonce":1431831900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6802","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6803","nonce":1431831900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6806","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6807","nonce":1431831900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6804","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6805","nonce":1431831900}]},"public_addr":"10.248.54.139:6801/1431831900","cluster_addr":"10.248.54.139:6803/1431831900","heartbeat_back_addr":"10.248.54.139:6807/1431831900","heartbeat_front_addr":"10.248.54.139:6805/1431831900","state":["exists","up"]},{"osd":4,"uuid":"6d593599-35ce-4b51-9b40-6ad4c83edfe8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6800","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6801","nonce":2521842530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6802","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6803","nonce":2521842530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6806","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6807","nonce":2521842530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6804","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6805","nonce":2521842530}]},"public_addr":"10.248.48.79:6801/2521842530","cluster_addr":"10.248.48.79:6803/2521842530","heartbeat_back_addr":"10.248.48.79:6807/2521842530","heartbeat_front_addr":"10.248.48.79:6805/2521842530","state":["exists","up"]},{"osd":5,"uuid":"aba968df-f3f1-439a-b6f2-5faa1ca225bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6800","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6801","nonce":3104603370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6802","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6803","nonce":3104603370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6806","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6807","nonce":3104603370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6804","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6805","nonce":3104603370}]},"public_addr":"10.248.48.80:6801/3104603370","cluster_addr":"10.248.48.80:6803/3104603370","heartbeat_back_addr":"10.248.48.80:6807/3104603370","heartbeat_front_addr":"10.248.48.80:6805/3104603370","state":["exists","up"]},{"osd":6,"uuid":"3669ed38-0c57-4ac7-be68-662766c3563f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6800","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6801","nonce":1566525895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6802","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6803","nonce":1566525895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6806","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6807","nonce":1566525895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6804","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6805","nonce":1566525895}]},"public_addr":"10.248.48.81:6801/1566525895","cluster_addr":"10.248.48.81:6803/1566525895","heartbeat_back_addr":"10.248.48.81:6807/1566525895","heartbeat_front_addr":"10.248.48.81:6805/1566525895","state":["exists","up"]},{"osd":7,"uuid":"2c41311d-2c04-4718-85bb-c1ab1b254771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6800","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6801","nonce":2451767405}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6802","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6803","nonce":2451767405}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6806","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6807","nonce":2451767405}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6804","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6805","nonce":2451767405}]},"public_addr":"10.248.50.79:6801/2451767405","cluster_addr":"10.248.50.79:6803/2451767405","heartbeat_back_addr":"10.248.50.79:6807/2451767405","heartbeat_front_addr":"10.248.50.79:6805/2451767405","state":["exists","up"]},{"osd":8,"uuid":"85334bda-c65b-46e7-bbdd-9f5e3a34f56e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6800","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6801","nonce":3495156876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6802","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6803","nonce":3495156876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6806","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6807","nonce":3495156876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6804","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6805","nonce":3495156876}]},"public_addr":"10.248.48.82:6801/3495156876","cluster_addr":"10.248.48.82:6803/3495156876","heartbeat_back_addr":"10.248.48.82:6807/3495156876","heartbeat_front_addr":"10.248.48.82:6805/3495156876","state":["exists","up"]},{"osd":9,"uuid":"4c44d120-5750-453c-9d7e-a4ec5e9ce827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6800","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6801","nonce":2897614832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6802","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6803","nonce":2897614832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6806","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6807","nonce":2897614832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6804","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6805","nonce":2897614832}]},"public_addr":"10.248.54.141:6801/2897614832","cluster_addr":"10.248.54.141:6803/2897614832","heartbeat_back_addr":"10.248.54.141:6807/2897614832","heartbeat_front_addr":"10.248.54.141:6805/2897614832","state":["exists","up"]},{"osd":10,"uuid":"33c06eb1-49c3-4ec9-8112-a26c3e402e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6800","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6801","nonce":986870350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6802","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6803","nonce":986870350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6806","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6807","nonce":986870350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6804","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6805","nonce":986870350}]},"public_addr":"10.248.50.80:6801/986870350","cluster_addr":"10.248.50.80:6803/986870350","heartbeat_back_addr":"10.248.50.80:6807/986870350","heartbeat_front_addr":"10.248.50.80:6805/986870350","state":["exists","up"]},{"osd":11,"uuid":"4db20cac-405b-4221-8711-8d3498c503dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6800","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6801","nonce":3203136589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6802","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6803","nonce":3203136589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6806","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6807","nonce":3203136589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6804","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6805","nonce":3203136589}]},"public_addr":"10.248.50.81:6801/3203136589","cluster_addr":"10.248.50.81:6803/3203136589","heartbeat_back_addr":"10.248.50.81:6807/3203136589","heartbeat_front_addr":"10.248.50.81:6805/3203136589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.014523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.982174+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.644667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.118715+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.398280+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:07.752303+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:09.049249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.907664+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:05.623192+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:13.439668+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:14.027346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:14.091177+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.50.72:6801/2669768639":"2022-03-07T00:24:16.422389+0000","10.248.50.72:6800/2669768639":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/267441637":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/3906748330":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/1481981300":"2022-03-07T00:24:16.422389+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-06T00:26:27.343 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-06T00:26:11.293986+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-06T00:26:27.344 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd pool get .mgr pg_num 2022-03-06T00:26:27.898 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2022-03-06T00:26:27.917 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-06T00:26:27.919 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-06T00:26:27.919 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph mgr dump --format=json 2022-03-06T00:26:28.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:28.549 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":12,"active_gid":14284,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.72:6800","nonce":227989611},{"type":"v1","addr":"10.248.50.72:6801","nonce":227989611}]},"active_addr":"10.248.50.72:6801/227989611","active_change":"2022-03-06T00:24:16.451246+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.248.50.72:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.248.50.72:0","nonce":2262325625}]},{"addrvec":[{"type":"v2","addr":"10.248.50.72:0","nonce":596784557}]},{"addrvec":[{"type":"v2","addr":"10.248.50.72:0","nonce":3144536037}]},{"addrvec":[{"type":"v2","addr":"10.248.50.72:0","nonce":3505395019}]}]}} 2022-03-06T00:26:28.552 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-06T00:26:28.553 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-06T00:26:28.553 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd dump --format=json 2022-03-06T00:26:29.063 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:29.063 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"b043b050-8999-48ec-a9a6-f3a813dbfc59","created":"2022-03-06T00:21:21.513204+0000","modified":"2022-03-06T00:26:18.089908+0000","last_up_change":"2022-03-06T00:26:15.925048+0000","last_in_change":"2022-03-06T00:25:32.853185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-06T00:26:11.293986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7e27b24f-261a-4450-9640-f4ba4019814d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6800","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6801","nonce":1924127139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6802","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6803","nonce":1924127139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6806","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6807","nonce":1924127139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6804","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6805","nonce":1924127139}]},"public_addr":"10.248.50.78:6801/1924127139","cluster_addr":"10.248.50.78:6803/1924127139","heartbeat_back_addr":"10.248.50.78:6807/1924127139","heartbeat_front_addr":"10.248.50.78:6805/1924127139","state":["exists","up"]},{"osd":1,"uuid":"7ba5e3e3-9fd0-4d84-bd54-acc99a222f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6800","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6801","nonce":1302588422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6802","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6803","nonce":1302588422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6806","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6807","nonce":1302588422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6804","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6805","nonce":1302588422}]},"public_addr":"10.248.54.138:6801/1302588422","cluster_addr":"10.248.54.138:6803/1302588422","heartbeat_back_addr":"10.248.54.138:6807/1302588422","heartbeat_front_addr":"10.248.54.138:6805/1302588422","state":["exists","up"]},{"osd":2,"uuid":"88598048-6d49-40db-8e1c-71dcd43867ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6800","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6801","nonce":924982503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6802","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6803","nonce":924982503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6806","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6807","nonce":924982503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6804","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6805","nonce":924982503}]},"public_addr":"10.248.54.140:6801/924982503","cluster_addr":"10.248.54.140:6803/924982503","heartbeat_back_addr":"10.248.54.140:6807/924982503","heartbeat_front_addr":"10.248.54.140:6805/924982503","state":["exists","up"]},{"osd":3,"uuid":"fd7c4212-ca13-4464-8fa2-de0e42305500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6800","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6801","nonce":1431831900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6802","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6803","nonce":1431831900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6806","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6807","nonce":1431831900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6804","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6805","nonce":1431831900}]},"public_addr":"10.248.54.139:6801/1431831900","cluster_addr":"10.248.54.139:6803/1431831900","heartbeat_back_addr":"10.248.54.139:6807/1431831900","heartbeat_front_addr":"10.248.54.139:6805/1431831900","state":["exists","up"]},{"osd":4,"uuid":"6d593599-35ce-4b51-9b40-6ad4c83edfe8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6800","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6801","nonce":2521842530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6802","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6803","nonce":2521842530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6806","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6807","nonce":2521842530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6804","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6805","nonce":2521842530}]},"public_addr":"10.248.48.79:6801/2521842530","cluster_addr":"10.248.48.79:6803/2521842530","heartbeat_back_addr":"10.248.48.79:6807/2521842530","heartbeat_front_addr":"10.248.48.79:6805/2521842530","state":["exists","up"]},{"osd":5,"uuid":"aba968df-f3f1-439a-b6f2-5faa1ca225bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6800","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6801","nonce":3104603370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6802","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6803","nonce":3104603370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6806","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6807","nonce":3104603370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6804","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6805","nonce":3104603370}]},"public_addr":"10.248.48.80:6801/3104603370","cluster_addr":"10.248.48.80:6803/3104603370","heartbeat_back_addr":"10.248.48.80:6807/3104603370","heartbeat_front_addr":"10.248.48.80:6805/3104603370","state":["exists","up"]},{"osd":6,"uuid":"3669ed38-0c57-4ac7-be68-662766c3563f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6800","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6801","nonce":1566525895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6802","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6803","nonce":1566525895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6806","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6807","nonce":1566525895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6804","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6805","nonce":1566525895}]},"public_addr":"10.248.48.81:6801/1566525895","cluster_addr":"10.248.48.81:6803/1566525895","heartbeat_back_addr":"10.248.48.81:6807/1566525895","heartbeat_front_addr":"10.248.48.81:6805/1566525895","state":["exists","up"]},{"osd":7,"uuid":"2c41311d-2c04-4718-85bb-c1ab1b254771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6800","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6801","nonce":2451767405}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6802","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6803","nonce":2451767405}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6806","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6807","nonce":2451767405}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6804","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6805","nonce":2451767405}]},"public_addr":"10.248.50.79:6801/2451767405","cluster_addr":"10.248.50.79:6803/2451767405","heartbeat_back_addr":"10.248.50.79:6807/2451767405","heartbeat_front_addr":"10.248.50.79:6805/2451767405","state":["exists","up"]},{"osd":8,"uuid":"85334bda-c65b-46e7-bbdd-9f5e3a34f56e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6800","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6801","nonce":3495156876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6802","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6803","nonce":3495156876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6806","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6807","nonce":3495156876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6804","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6805","nonce":3495156876}]},"public_addr":"10.248.48.82:6801/3495156876","cluster_addr":"10.248.48.82:6803/3495156876","heartbeat_back_addr":"10.248.48.82:6807/3495156876","heartbeat_front_addr":"10.248.48.82:6805/3495156876","state":["exists","up"]},{"osd":9,"uuid":"4c44d120-5750-453c-9d7e-a4ec5e9ce827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6800","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6801","nonce":2897614832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6802","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6803","nonce":2897614832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6806","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6807","nonce":2897614832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6804","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6805","nonce":2897614832}]},"public_addr":"10.248.54.141:6801/2897614832","cluster_addr":"10.248.54.141:6803/2897614832","heartbeat_back_addr":"10.248.54.141:6807/2897614832","heartbeat_front_addr":"10.248.54.141:6805/2897614832","state":["exists","up"]},{"osd":10,"uuid":"33c06eb1-49c3-4ec9-8112-a26c3e402e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6800","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6801","nonce":986870350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6802","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6803","nonce":986870350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6806","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6807","nonce":986870350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6804","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6805","nonce":986870350}]},"public_addr":"10.248.50.80:6801/986870350","cluster_addr":"10.248.50.80:6803/986870350","heartbeat_back_addr":"10.248.50.80:6807/986870350","heartbeat_front_addr":"10.248.50.80:6805/986870350","state":["exists","up"]},{"osd":11,"uuid":"4db20cac-405b-4221-8711-8d3498c503dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6800","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6801","nonce":3203136589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6802","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6803","nonce":3203136589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6806","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6807","nonce":3203136589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6804","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6805","nonce":3203136589}]},"public_addr":"10.248.50.81:6801/3203136589","cluster_addr":"10.248.50.81:6803/3203136589","heartbeat_back_addr":"10.248.50.81:6807/3203136589","heartbeat_front_addr":"10.248.50.81:6805/3203136589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.014523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.982174+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.644667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.118715+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.398280+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:07.752303+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:09.049249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.907664+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:05.623192+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:13.439668+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:14.027346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:14.091177+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.50.72:6801/2669768639":"2022-03-07T00:24:16.422389+0000","10.248.50.72:6800/2669768639":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/267441637":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/3906748330":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/1481981300":"2022-03-07T00:24:16.422389+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-06T00:26:29.083 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-06T00:26:29.083 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd dump --format=json 2022-03-06T00:26:29.609 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:29.610 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"b043b050-8999-48ec-a9a6-f3a813dbfc59","created":"2022-03-06T00:21:21.513204+0000","modified":"2022-03-06T00:26:18.089908+0000","last_up_change":"2022-03-06T00:26:15.925048+0000","last_in_change":"2022-03-06T00:25:32.853185+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-06T00:26:11.293986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7e27b24f-261a-4450-9640-f4ba4019814d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6800","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6801","nonce":1924127139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6802","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6803","nonce":1924127139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6806","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6807","nonce":1924127139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.78:6804","nonce":1924127139},{"type":"v1","addr":"10.248.50.78:6805","nonce":1924127139}]},"public_addr":"10.248.50.78:6801/1924127139","cluster_addr":"10.248.50.78:6803/1924127139","heartbeat_back_addr":"10.248.50.78:6807/1924127139","heartbeat_front_addr":"10.248.50.78:6805/1924127139","state":["exists","up"]},{"osd":1,"uuid":"7ba5e3e3-9fd0-4d84-bd54-acc99a222f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6800","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6801","nonce":1302588422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6802","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6803","nonce":1302588422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6806","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6807","nonce":1302588422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.138:6804","nonce":1302588422},{"type":"v1","addr":"10.248.54.138:6805","nonce":1302588422}]},"public_addr":"10.248.54.138:6801/1302588422","cluster_addr":"10.248.54.138:6803/1302588422","heartbeat_back_addr":"10.248.54.138:6807/1302588422","heartbeat_front_addr":"10.248.54.138:6805/1302588422","state":["exists","up"]},{"osd":2,"uuid":"88598048-6d49-40db-8e1c-71dcd43867ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6800","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6801","nonce":924982503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6802","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6803","nonce":924982503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6806","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6807","nonce":924982503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.140:6804","nonce":924982503},{"type":"v1","addr":"10.248.54.140:6805","nonce":924982503}]},"public_addr":"10.248.54.140:6801/924982503","cluster_addr":"10.248.54.140:6803/924982503","heartbeat_back_addr":"10.248.54.140:6807/924982503","heartbeat_front_addr":"10.248.54.140:6805/924982503","state":["exists","up"]},{"osd":3,"uuid":"fd7c4212-ca13-4464-8fa2-de0e42305500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6800","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6801","nonce":1431831900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6802","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6803","nonce":1431831900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6806","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6807","nonce":1431831900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.139:6804","nonce":1431831900},{"type":"v1","addr":"10.248.54.139:6805","nonce":1431831900}]},"public_addr":"10.248.54.139:6801/1431831900","cluster_addr":"10.248.54.139:6803/1431831900","heartbeat_back_addr":"10.248.54.139:6807/1431831900","heartbeat_front_addr":"10.248.54.139:6805/1431831900","state":["exists","up"]},{"osd":4,"uuid":"6d593599-35ce-4b51-9b40-6ad4c83edfe8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6800","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6801","nonce":2521842530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6802","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6803","nonce":2521842530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6806","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6807","nonce":2521842530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.79:6804","nonce":2521842530},{"type":"v1","addr":"10.248.48.79:6805","nonce":2521842530}]},"public_addr":"10.248.48.79:6801/2521842530","cluster_addr":"10.248.48.79:6803/2521842530","heartbeat_back_addr":"10.248.48.79:6807/2521842530","heartbeat_front_addr":"10.248.48.79:6805/2521842530","state":["exists","up"]},{"osd":5,"uuid":"aba968df-f3f1-439a-b6f2-5faa1ca225bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6800","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6801","nonce":3104603370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6802","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6803","nonce":3104603370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6806","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6807","nonce":3104603370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.80:6804","nonce":3104603370},{"type":"v1","addr":"10.248.48.80:6805","nonce":3104603370}]},"public_addr":"10.248.48.80:6801/3104603370","cluster_addr":"10.248.48.80:6803/3104603370","heartbeat_back_addr":"10.248.48.80:6807/3104603370","heartbeat_front_addr":"10.248.48.80:6805/3104603370","state":["exists","up"]},{"osd":6,"uuid":"3669ed38-0c57-4ac7-be68-662766c3563f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6800","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6801","nonce":1566525895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6802","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6803","nonce":1566525895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6806","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6807","nonce":1566525895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.81:6804","nonce":1566525895},{"type":"v1","addr":"10.248.48.81:6805","nonce":1566525895}]},"public_addr":"10.248.48.81:6801/1566525895","cluster_addr":"10.248.48.81:6803/1566525895","heartbeat_back_addr":"10.248.48.81:6807/1566525895","heartbeat_front_addr":"10.248.48.81:6805/1566525895","state":["exists","up"]},{"osd":7,"uuid":"2c41311d-2c04-4718-85bb-c1ab1b254771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6800","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6801","nonce":2451767405}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6802","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6803","nonce":2451767405}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6806","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6807","nonce":2451767405}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.79:6804","nonce":2451767405},{"type":"v1","addr":"10.248.50.79:6805","nonce":2451767405}]},"public_addr":"10.248.50.79:6801/2451767405","cluster_addr":"10.248.50.79:6803/2451767405","heartbeat_back_addr":"10.248.50.79:6807/2451767405","heartbeat_front_addr":"10.248.50.79:6805/2451767405","state":["exists","up"]},{"osd":8,"uuid":"85334bda-c65b-46e7-bbdd-9f5e3a34f56e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6800","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6801","nonce":3495156876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6802","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6803","nonce":3495156876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6806","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6807","nonce":3495156876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.48.82:6804","nonce":3495156876},{"type":"v1","addr":"10.248.48.82:6805","nonce":3495156876}]},"public_addr":"10.248.48.82:6801/3495156876","cluster_addr":"10.248.48.82:6803/3495156876","heartbeat_back_addr":"10.248.48.82:6807/3495156876","heartbeat_front_addr":"10.248.48.82:6805/3495156876","state":["exists","up"]},{"osd":9,"uuid":"4c44d120-5750-453c-9d7e-a4ec5e9ce827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6800","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6801","nonce":2897614832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6802","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6803","nonce":2897614832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6806","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6807","nonce":2897614832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.54.141:6804","nonce":2897614832},{"type":"v1","addr":"10.248.54.141:6805","nonce":2897614832}]},"public_addr":"10.248.54.141:6801/2897614832","cluster_addr":"10.248.54.141:6803/2897614832","heartbeat_back_addr":"10.248.54.141:6807/2897614832","heartbeat_front_addr":"10.248.54.141:6805/2897614832","state":["exists","up"]},{"osd":10,"uuid":"33c06eb1-49c3-4ec9-8112-a26c3e402e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6800","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6801","nonce":986870350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6802","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6803","nonce":986870350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6806","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6807","nonce":986870350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.80:6804","nonce":986870350},{"type":"v1","addr":"10.248.50.80:6805","nonce":986870350}]},"public_addr":"10.248.50.80:6801/986870350","cluster_addr":"10.248.50.80:6803/986870350","heartbeat_back_addr":"10.248.50.80:6807/986870350","heartbeat_front_addr":"10.248.50.80:6805/986870350","state":["exists","up"]},{"osd":11,"uuid":"4db20cac-405b-4221-8711-8d3498c503dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6800","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6801","nonce":3203136589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6802","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6803","nonce":3203136589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6806","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6807","nonce":3203136589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.248.50.81:6804","nonce":3203136589},{"type":"v1","addr":"10.248.50.81:6805","nonce":3203136589}]},"public_addr":"10.248.50.81:6801/3203136589","cluster_addr":"10.248.50.81:6803/3203136589","heartbeat_back_addr":"10.248.50.81:6807/3203136589","heartbeat_front_addr":"10.248.50.81:6805/3203136589","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.014523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.982174+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.644667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.118715+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:01.398280+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:07.752303+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:09.049249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:10.907664+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:05.623192+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:13.439668+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:14.027346+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-06T00:26:14.091177+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.248.50.72:6801/2669768639":"2022-03-07T00:24:16.422389+0000","10.248.50.72:6800/2669768639":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/267441637":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/3906748330":"2022-03-07T00:24:16.422389+0000","10.248.50.72:0/1481981300":"2022-03-07T00:24:16.422389+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-06T00:26:29.629 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.0 flush_pg_stats 2022-03-06T00:26:29.630 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.1 flush_pg_stats 2022-03-06T00:26:29.630 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.2 flush_pg_stats 2022-03-06T00:26:29.631 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.3 flush_pg_stats 2022-03-06T00:26:29.631 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.4 flush_pg_stats 2022-03-06T00:26:29.632 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.5 flush_pg_stats 2022-03-06T00:26:29.633 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.6 flush_pg_stats 2022-03-06T00:26:29.633 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.7 flush_pg_stats 2022-03-06T00:26:29.634 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.8 flush_pg_stats 2022-03-06T00:26:29.635 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.9 flush_pg_stats 2022-03-06T00:26:29.636 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.10 flush_pg_stats 2022-03-06T00:26:29.636 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph tell osd.11 flush_pg_stats 2022-03-06T00:26:30.141 INFO:teuthology.orchestra.run.smithi007.stdout:73014444039 2022-03-06T00:26:30.142 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.0 2022-03-06T00:26:30.296 INFO:teuthology.orchestra.run.smithi007.stdout:94489280519 2022-03-06T00:26:30.297 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.7 2022-03-06T00:26:30.675 INFO:teuthology.orchestra.run.smithi007.stdout:77309411335 2022-03-06T00:26:30.676 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.4 2022-03-06T00:26:30.742 INFO:teuthology.orchestra.run.smithi007.stdout:94489280518 2022-03-06T00:26:30.743 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.6 2022-03-06T00:26:30.751 INFO:teuthology.orchestra.run.smithi007.stdout:94489280518 2022-03-06T00:26:30.752 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.3 2022-03-06T00:26:30.754 INFO:teuthology.orchestra.run.smithi007.stdout:85899345926 2022-03-06T00:26:30.755 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.5 2022-03-06T00:26:30.758 INFO:teuthology.orchestra.run.smithi007.stdout:94489280517 2022-03-06T00:26:30.759 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.2 2022-03-06T00:26:30.767 INFO:teuthology.orchestra.run.smithi007.stdout:107374182405 2022-03-06T00:26:30.768 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.9 2022-03-06T00:26:30.770 INFO:teuthology.orchestra.run.smithi007.stdout:85899345926 2022-03-06T00:26:30.771 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.8 2022-03-06T00:26:30.846 INFO:teuthology.orchestra.run.smithi007.stdout:107374182405 2022-03-06T00:26:30.846 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.11 2022-03-06T00:26:30.857 INFO:teuthology.orchestra.run.smithi007.stdout:107374182406 2022-03-06T00:26:30.857 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.10 2022-03-06T00:26:30.862 INFO:teuthology.orchestra.run.smithi007.stdout:81604378632 2022-03-06T00:26:30.862 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.1 2022-03-06T00:26:31.217 INFO:teuthology.orchestra.run.smithi007.stdout:94489280519 2022-03-06T00:26:31.286 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280519 got 94489280519 for osd.7 2022-03-06T00:26:31.288 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:31.298 INFO:teuthology.orchestra.run.smithi007.stdout:73014444039 2022-03-06T00:26:31.378 INFO:teuthology.orchestra.run.smithi007.stdout:77309411335 2022-03-06T00:26:31.550 INFO:teuthology.orchestra.run.smithi007.stdout:94489280518 2022-03-06T00:26:31.597 INFO:tasks.rook.ceph_manager.ceph:need seq 73014444039 got 73014444039 for osd.0 2022-03-06T00:26:31.598 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:31.600 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411335 got 77309411335 for osd.4 2022-03-06T00:26:31.601 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:31.607 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280518 got 94489280518 for osd.6 2022-03-06T00:26:31.608 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:31.620 INFO:teuthology.orchestra.run.smithi007.stdout:85899345926 2022-03-06T00:26:31.649 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345926 got 85899345926 for osd.8 2022-03-06T00:26:31.649 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:31.699 INFO:teuthology.orchestra.run.smithi007.stdout:85899345926 2022-03-06T00:26:31.739 INFO:teuthology.orchestra.run.smithi007.stdout:94489280518 2022-03-06T00:26:31.745 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345926 got 85899345926 for osd.5 2022-03-06T00:26:31.745 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:31.771 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280518 got 94489280518 for osd.3 2022-03-06T00:26:31.771 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:31.859 INFO:teuthology.orchestra.run.smithi007.stdout:107374182405 2022-03-06T00:26:31.886 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182406 got 107374182405 for osd.10 2022-03-06T00:26:31.900 INFO:teuthology.orchestra.run.smithi007.stdout:94489280517 2022-03-06T00:26:31.926 INFO:tasks.rook.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.2 2022-03-06T00:26:31.927 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:31.985 INFO:teuthology.orchestra.run.smithi007.stdout:107374182405 2022-03-06T00:26:32.006 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182405 got 107374182405 for osd.11 2022-03-06T00:26:32.006 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:32.054 INFO:teuthology.orchestra.run.smithi007.stdout:107374182405 2022-03-06T00:26:32.079 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182405 got 107374182405 for osd.9 2022-03-06T00:26:32.080 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:32.123 INFO:teuthology.orchestra.run.smithi007.stdout:81604378631 2022-03-06T00:26:32.143 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378632 got 81604378631 for osd.1 2022-03-06T00:26:32.888 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.10 2022-03-06T00:26:33.144 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.1 2022-03-06T00:26:33.425 INFO:teuthology.orchestra.run.smithi007.stdout:107374182405 2022-03-06T00:26:33.445 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182406 got 107374182405 for osd.10 2022-03-06T00:26:33.686 INFO:teuthology.orchestra.run.smithi007.stdout:81604378632 2022-03-06T00:26:33.708 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378632 got 81604378632 for osd.1 2022-03-06T00:26:33.708 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:34.446 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph osd last-stat-seq osd.10 2022-03-06T00:26:34.947 INFO:teuthology.orchestra.run.smithi007.stdout:107374182407 2022-03-06T00:26:34.967 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182406 got 107374182407 for osd.10 2022-03-06T00:26:34.967 DEBUG:teuthology.parallel:result is None 2022-03-06T00:26:34.968 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-06T00:26:34.968 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph pg dump --format=json 2022-03-06T00:26:35.523 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2022-03-06T00:26:35.543 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2022-03-06T00:26:34.770947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":64888,"kb_used_data":4632,"kb_used_omap":0,"kb_used_meta":59776,"kb_avail":1124876936,"statfs":{"total":1151940427776,"available":1151873982464,"internally_reserved":0,"allocated":4743168,"data_stored":2852784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61210624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003791"},"pg_stats":[{"pgid":"1.0","version":"27'137","reported_seq":188,"reported_epoch":27,"state":"active+clean","last_fresh":"2022-03-06T00:26:26.032013+0000","last_change":"2022-03-06T00:26:13.964179+0000","last_active":"2022-03-06T00:26:26.032013+0000","last_peered":"2022-03-06T00:26:26.032013+0000","last_clean":"2022-03-06T00:26:26.032013+0000","last_became_active":"2022-03-06T00:26:13.962871+0000","last_became_peered":"2022-03-06T00:26:13.962871+0000","last_unstale":"2022-03-06T00:26:26.032013+0000","last_undegraded":"2022-03-06T00:26:26.032013+0000","last_fullsized":"2022-03-06T00:26:26.032013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-06T00:26:11.311361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-06T00:26:11.311361+0000","last_clean_scrub_stamp":"2022-03-06T00:26:11.311361+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-07T06:17:17.812092+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":25,"seq":107374182406,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3919999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8759999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4039999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1190000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0089999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1890000000000001}]}]},{"osd":10,"up_from":25,"seq":107374182407,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9219999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.762}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0249999999999999}]}]},{"osd":9,"up_from":25,"seq":107374182406,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7210000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.29}]}]},{"osd":7,"up_from":22,"seq":94489280520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.821}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7250000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.129}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.274}]}]},{"osd":6,"up_from":22,"seq":94489280519,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":794624,"data_stored":634428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3610000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4220000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.911}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.516}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.915}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8580000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3090000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2210000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5569999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5019999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8420000000000001}]}]},{"osd":4,"up_from":18,"seq":77309411336,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]}]},{"osd":0,"up_from":17,"seq":73014444040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.732}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]}]},{"osd":1,"up_from":19,"seq":81604378633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5579999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4540000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1709999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.073}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7639999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1389999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3290000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8879999999999999}]}]},{"osd":8,"up_from":20,"seq":85899345927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":794624,"data_stored":634428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4510000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]}]},{"osd":5,"up_from":20,"seq":85899345927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]}]},{"osd":2,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":794624,"data_stored":634428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9770000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.742}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5259999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.573}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6659999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-06T00:26:35.545 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph pg dump --format=json 2022-03-06T00:26:36.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:36.064 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2022-03-06T00:26:36.083 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2022-03-06T00:26:34.770947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":64888,"kb_used_data":4632,"kb_used_omap":0,"kb_used_meta":59776,"kb_avail":1124876936,"statfs":{"total":1151940427776,"available":1151873982464,"internally_reserved":0,"allocated":4743168,"data_stored":2852784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61210624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003791"},"pg_stats":[{"pgid":"1.0","version":"27'137","reported_seq":188,"reported_epoch":27,"state":"active+clean","last_fresh":"2022-03-06T00:26:26.032013+0000","last_change":"2022-03-06T00:26:13.964179+0000","last_active":"2022-03-06T00:26:26.032013+0000","last_peered":"2022-03-06T00:26:26.032013+0000","last_clean":"2022-03-06T00:26:26.032013+0000","last_became_active":"2022-03-06T00:26:13.962871+0000","last_became_peered":"2022-03-06T00:26:13.962871+0000","last_unstale":"2022-03-06T00:26:26.032013+0000","last_undegraded":"2022-03-06T00:26:26.032013+0000","last_fullsized":"2022-03-06T00:26:26.032013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-06T00:26:11.311361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-06T00:26:11.311361+0000","last_clean_scrub_stamp":"2022-03-06T00:26:11.311361+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-07T06:17:17.812092+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":25,"seq":107374182406,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3919999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.952}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8759999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4039999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1190000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0089999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1890000000000001}]}]},{"osd":10,"up_from":25,"seq":107374182407,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9219999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.762}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0249999999999999}]}]},{"osd":9,"up_from":25,"seq":107374182406,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7210000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.29}]}]},{"osd":7,"up_from":22,"seq":94489280520,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.821}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7250000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.129}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.274}]}]},{"osd":6,"up_from":22,"seq":94489280519,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":794624,"data_stored":634428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3610000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4220000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.911}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.516}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.915}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8580000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740088,"statfs":{"total":95995035648,"available":95989850112,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3090000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2210000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5569999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5019999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8420000000000001}]}]},{"osd":4,"up_from":18,"seq":77309411336,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]}]},{"osd":0,"up_from":17,"seq":73014444040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.732}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.865}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6830000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5780000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8049999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]}]},{"osd":1,"up_from":19,"seq":81604378633,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5579999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4540000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1709999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.073}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7639999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1389999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3290000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8879999999999999}]}]},{"osd":8,"up_from":20,"seq":85899345927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":794624,"data_stored":634428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4510000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]}]},{"osd":5,"up_from":20,"seq":85899345927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5128,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93740024,"statfs":{"total":95995035648,"available":95989784576,"internally_reserved":0,"allocated":262144,"data_stored":105500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]}]},{"osd":2,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":776,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":794624,"data_stored":634428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1869999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9770000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9350000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.742}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5259999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.573}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5049999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6659999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-06T00:26:36.085 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-06T00:26:36.086 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-06T00:26:36.086 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-06T00:26:36.086 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph health --format=json 2022-03-06T00:26:36.645 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:36.964 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-06T00:26:36.965 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-06T00:26:36.966 INFO:tasks.rook:Rook complete, yielding 2022-03-06T00:26:36.966 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-06T00:26:36.982 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph -s 2022-03-06T00:26:37.584 INFO:teuthology.orchestra.run.smithi007.stdout: cluster: 2022-03-06T00:26:37.585 INFO:teuthology.orchestra.run.smithi007.stdout: id: b043b050-8999-48ec-a9a6-f3a813dbfc59 2022-03-06T00:26:37.585 INFO:teuthology.orchestra.run.smithi007.stdout: health: HEALTH_OK 2022-03-06T00:26:37.585 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:37.586 INFO:teuthology.orchestra.run.smithi007.stdout: services: 2022-03-06T00:26:37.586 INFO:teuthology.orchestra.run.smithi007.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-06T00:26:37.586 INFO:teuthology.orchestra.run.smithi007.stdout: mgr: a(active, since 2m) 2022-03-06T00:26:37.586 INFO:teuthology.orchestra.run.smithi007.stdout: osd: 12 osds: 12 up (since 21s), 12 in (since 64s) 2022-03-06T00:26:37.587 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:37.587 INFO:teuthology.orchestra.run.smithi007.stdout: data: 2022-03-06T00:26:37.587 INFO:teuthology.orchestra.run.smithi007.stdout: pools: 1 pools, 1 pgs 2022-03-06T00:26:37.587 INFO:teuthology.orchestra.run.smithi007.stdout: objects: 2 objects, 577 KiB 2022-03-06T00:26:37.588 INFO:teuthology.orchestra.run.smithi007.stdout: usage: 63 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-06T00:26:37.588 INFO:teuthology.orchestra.run.smithi007.stdout: pgs: 1 active+clean 2022-03-06T00:26:37.588 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:37.588 INFO:teuthology.orchestra.run.smithi007.stdout: io: 2022-03-06T00:26:37.588 INFO:teuthology.orchestra.run.smithi007.stdout: client: 767 B/s rd, 22 KiB/s wr, 0 op/s rd, 1 op/s wr 2022-03-06T00:26:37.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:26:37.607 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status 2022-03-06T00:26:38.352 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: rook 2022-03-06T00:26:38.352 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2022-03-06T00:26:38.376 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ps 2022-03-06T00:26:38.959 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-06T00:26:38.959 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi007 running (2m) 0s ago 2m - - 02136b44099a 2022-03-06T00:26:38.959 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi017 running (2m) 0s ago 2m - - 02136b44099a 2022-03-06T00:26:38.960 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi066 running (76s) 0s ago 76s - - 02136b44099a 2022-03-06T00:26:38.960 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.a smithi066 running (2m) 0s ago 2m - - 02136b44099a 2022-03-06T00:26:38.960 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi066 running (5m) 0s ago 5m - - 02136b44099a 2022-03-06T00:26:38.960 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi017 running (4m) 0s ago 4m - - 02136b44099a 2022-03-06T00:26:38.961 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (4m) 0s ago 4m - - 02136b44099a 2022-03-06T00:26:38.961 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi066 running (76s) 0s ago 76s - - 02136b44099a 2022-03-06T00:26:38.961 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (73s) 0s ago 73s - - 02136b44099a 2022-03-06T00:26:38.961 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 smithi066 running (57s) 0s ago 57s - - 02136b44099a 2022-03-06T00:26:38.961 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 smithi066 running (55s) 0s ago 55s - - 02136b44099a 2022-03-06T00:26:38.962 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (67s) 0s ago 68s - - 02136b44099a 2022-03-06T00:26:38.962 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (68s) 0s ago 69s - - 02136b44099a 2022-03-06T00:26:38.962 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi017 running (68s) 0s ago 68s - - 02136b44099a 2022-03-06T00:26:38.962 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi017 running (64s) 0s ago 64s - - 02136b44099a 2022-03-06T00:26:38.963 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi017 running (63s) 0s ago 63s - - 02136b44099a 2022-03-06T00:26:38.963 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi066 running (61s) 0s ago 61s - - 02136b44099a 2022-03-06T00:26:38.963 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 smithi017 running (60s) 0s ago 61s - - 02136b44099a 2022-03-06T00:26:38.963 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 smithi007 running (59s) 0s ago 59s - - 02136b44099a 2022-03-06T00:26:38.983 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls 2022-03-06T00:26:39.902 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-06T00:26:39.903 INFO:teuthology.orchestra.run.smithi007.stdout:crash 3/3 0s ago 2m * 2022-03-06T00:26:39.903 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 1/1 0s ago 2m count:1 2022-03-06T00:26:39.903 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 0s ago 5m count:3 2022-03-06T00:26:39.903 INFO:teuthology.orchestra.run.smithi007.stdout:osd 24 0s ago 77s 2022-03-06T00:26:39.903 INFO:teuthology.orchestra.run.smithi007.stdout:osd.all-available-devices 0 0s ago - * 2022-03-06T00:26:39.962 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch device ls 2022-03-06T00:26:40.541 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-06T00:26:40.542 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:26:40.542 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:26:40.543 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:26:40.543 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:26:40.544 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:26:40.544 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:26:40.545 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:26:40.545 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:26:40.545 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:26:40.546 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:26:40.546 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:26:40.547 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:26:40.566 INFO:teuthology.run_tasks:Running task exec... 2022-03-06T00:26:40.579 INFO:teuthology.task.exec:Executing custom commands... 2022-03-06T00:26:40.580 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi007.front.sepia.ceph.com 2022-03-06T00:26:40.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-06T00:26:40.582 DEBUG:teuthology.orchestra.run.smithi007:> toolbox() { 2022-03-06T00:26:40.582 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-06T00:26:40.583 DEBUG:teuthology.orchestra.run.smithi007:> } 2022-03-06T00:26:40.583 DEBUG:teuthology.orchestra.run.smithi007:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-06T00:26:40.584 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch osd rm 0 --force 2022-03-06T00:26:40.584 DEBUG:teuthology.orchestra.run.smithi007:> removed_pv="" 2022-03-06T00:26:40.585 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$removed_pv" = "" ] 2022-03-06T00:26:40.586 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-06T00:26:40.586 DEBUG:teuthology.orchestra.run.smithi007:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-06T00:26:40.587 DEBUG:teuthology.orchestra.run.smithi007:> sleep 3s 2022-03-06T00:26:40.587 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-06T00:26:40.588 DEBUG:teuthology.orchestra.run.smithi007:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-06T00:26:40.588 DEBUG:teuthology.orchestra.run.smithi007:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-06T00:26:40.589 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch device zap $host $target_path --force 2022-03-06T00:26:40.589 DEBUG:teuthology.orchestra.run.smithi007:> zap_completion="0" 2022-03-06T00:26:40.590 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$zap_completion" = "0" ] 2022-03-06T00:26:40.591 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-06T00:26:40.591 DEBUG:teuthology.orchestra.run.smithi007:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-06T00:26:40.592 DEBUG:teuthology.orchestra.run.smithi007:> sleep 3s 2022-03-06T00:26:40.592 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-06T00:26:40.594 DEBUG:teuthology.orchestra.run.smithi007:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-06T00:26:40.594 DEBUG:teuthology.orchestra.run.smithi007:> toolbox ceph orch apply osd --all-available-devices 2022-03-06T00:26:40.594 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-06T00:26:40.595 DEBUG:teuthology.orchestra.run.smithi007:> num_osd="0" 2022-03-06T00:26:40.595 DEBUG:teuthology.orchestra.run.smithi007:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-06T00:26:40.596 DEBUG:teuthology.orchestra.run.smithi007:> do 2022-03-06T00:26:40.596 DEBUG:teuthology.orchestra.run.smithi007:> echo "waiting for osd to come back up" 2022-03-06T00:26:40.596 DEBUG:teuthology.orchestra.run.smithi007:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-06T00:26:40.596 DEBUG:teuthology.orchestra.run.smithi007:> sleep 30s 2022-03-06T00:26:40.597 DEBUG:teuthology.orchestra.run.smithi007:> done 2022-03-06T00:26:40.597 DEBUG:teuthology.orchestra.run.smithi007:> ' 2022-03-06T00:26:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-06T00:26:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-06T00:26:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:26:42.916 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:26:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:+ orig_num_osd=12 2022-03-06T00:26:43.405 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-06T00:26:43.406 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-06T00:26:43.483 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:26:44.164 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.156615 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-06T00:26:44.164 INFO:tasks.rook.operator.smithi007.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", AllowUnsupported: true}, 2022-03-06T00:26:44.165 INFO:tasks.rook.operator.smithi007.stdout: Storage: v1.StorageScopeSpec{ 2022-03-06T00:26:44.165 INFO:tasks.rook.operator.smithi007.stdout: ... // 3 identical fields 2022-03-06T00:26:44.165 INFO:tasks.rook.operator.smithi007.stdout: Config: nil, 2022-03-06T00:26:44.165 INFO:tasks.rook.operator.smithi007.stdout: Selection: {}, 2022-03-06T00:26:44.165 INFO:tasks.rook.operator.smithi007.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-06T00:26:44.166 INFO:tasks.rook.operator.smithi007.stdout: ... // 9 identical elements 2022-03-06T00:26:44.166 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi066-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi066-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:26:44.166 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi066-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi066-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:26:44.166 INFO:tasks.rook.operator.smithi007.stdout:- { 2022-03-06T00:26:44.167 INFO:tasks.rook.operator.smithi007.stdout:- Name: "smithi066-lv-4", 2022-03-06T00:26:44.167 INFO:tasks.rook.operator.smithi007.stdout:- Count: 1, 2022-03-06T00:26:44.167 INFO:tasks.rook.operator.smithi007.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-06T00:26:44.167 INFO:tasks.rook.operator.smithi007.stdout:- }, 2022-03-06T00:26:44.168 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-06T00:26:44.168 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-06T00:26:44.168 INFO:tasks.rook.operator.smithi007.stdout: Annotations: nil, 2022-03-06T00:26:44.168 INFO:tasks.rook.operator.smithi007.stdout: Labels: nil, 2022-03-06T00:26:44.169 INFO:tasks.rook.operator.smithi007.stdout: ... // 20 identical fields 2022-03-06T00:26:44.169 INFO:tasks.rook.operator.smithi007.stdout: } 2022-03-06T00:26:44.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.156754 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-06T00:26:44.169 INFO:tasks.rook.operator.smithi007.stdout:I0306 00:26:44.157059 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-06T00:26:44.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.157178 I | operator: successfully started the controller-runtime manager 2022-03-06T00:26:44.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.186637 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:26:44.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.186666 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:26:44.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.186708 I | operator: setting up schemes 2022-03-06T00:26:44.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.191491 I | operator: setting up the controller-runtime manager 2022-03-06T00:26:44.499 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.494697 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:26:44.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539199 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:26:44.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539252 I | ceph-cluster-controller: successfully started 2022-03-06T00:26:44.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539345 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:26:44.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539381 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:26:44.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539426 I | ceph-block-pool-controller: successfully started 2022-03-06T00:26:44.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539470 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:26:44.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539519 I | ceph-object-realm-controller: successfully started 2022-03-06T00:26:44.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539556 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:26:44.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539581 I | ceph-object-zone-controller: successfully started 2022-03-06T00:26:44.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539832 I | ceph-object-controller: successfully started 2022-03-06T00:26:44.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539911 I | ceph-file-controller: successfully started 2022-03-06T00:26:44.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.539979 I | ceph-nfs-controller: successfully started 2022-03-06T00:26:44.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540035 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:26:44.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540081 I | ceph-client-controller: successfully started 2022-03-06T00:26:44.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540122 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:26:44.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540172 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:26:44.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540211 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:26:44.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540246 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:26:44.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540282 I | ceph-bucket-topic: successfully started 2022-03-06T00:26:44.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540306 I | ceph-bucket-notification: successfully started 2022-03-06T00:26:44.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540334 I | ceph-bucket-notification: successfully started 2022-03-06T00:26:44.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.540359 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:26:44.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.542015 I | operator: starting the controller-runtime manager 2022-03-06T00:26:44.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.647677 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:26:44.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.647731 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:26:44.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.647744 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:26:44.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.647817 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:26:44.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.699765 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:26:44.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.702478 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:26:44.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.704129 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:26:44.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.704208 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-06T00:26:44.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.704226 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:26:44.713 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.707795 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-06T00:26:44.713 INFO:tasks.rook.operator.smithi007.stdout:I0306 00:26:44.708118 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:26:44.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.722435 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-06T00:26:44.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.777400 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:26:44.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.777989 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-06T00:26:44.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778011 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-06T00:26:44.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778020 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-06T00:26:44.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778055 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-06T00:26:44.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778069 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-06T00:26:44.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778081 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-06T00:26:44.791 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778101 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-06T00:26:44.791 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778113 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-06T00:26:44.791 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778121 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-06T00:26:44.791 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778133 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-06T00:26:44.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778146 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-06T00:26:44.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778162 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-06T00:26:44.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778176 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-06T00:26:44.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778183 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-06T00:26:44.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778190 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-06T00:26:44.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778200 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-06T00:26:44.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778273 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:26:44.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:44.778297 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:26:47.436 INFO:teuthology.orchestra.run.smithi007.stdout:Success 2022-03-06T00:26:47.466 INFO:teuthology.orchestra.run.smithi007.stderr:+ removed_pv= 2022-03-06T00:26:47.467 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' '' = '' ']' 2022-03-06T00:26:47.467 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get pv 2022-03-06T00:26:47.467 INFO:teuthology.orchestra.run.smithi007.stderr:++ grep Released 2022-03-06T00:26:47.467 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f1 -d ' ' 2022-03-06T00:26:47.605 INFO:teuthology.orchestra.run.smithi007.stderr:+ removed_pv= 2022-03-06T00:26:47.605 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 3s 2022-03-06T00:26:47.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:47.815864 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-06T00:26:50.608 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' '' = '' ']' 2022-03-06T00:26:50.609 INFO:teuthology.orchestra.run.smithi007.stderr:++ grep Released 2022-03-06T00:26:50.610 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get pv 2022-03-06T00:26:50.610 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f1 -d ' ' 2022-03-06T00:26:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:+ removed_pv=smithi066-lv-4 2022-03-06T00:26:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 3s 2022-03-06T00:26:52.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636429 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-06T00:26:52.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636486 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-06T00:26:52.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636502 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:26:52.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636509 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:26:52.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636517 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:26:52.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636527 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:26:52.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636545 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:26:52.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636562 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:26:52.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636580 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:26:52.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636598 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:26:52.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636612 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:26:52.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636624 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:26:52.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636640 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:26:52.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636658 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:26:52.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636675 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-06T00:26:52.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636701 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-06T00:26:52.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636711 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-06T00:26:52.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636729 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-06T00:26:52.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636747 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-06T00:26:52.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636774 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:26:52.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636783 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:26:52.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636789 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-06T00:26:52.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636791 I | ceph-csi: Kubernetes version is 1.23 2022-03-06T00:26:52.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636795 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-06T00:26:52.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.636798 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-06T00:26:52.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.640386 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-06T00:26:52.657 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.649714 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:26:52.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.649740 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:26:52.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.649744 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:26:52.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.649748 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:26:52.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.649751 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:26:52.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.649755 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:26:52.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.649760 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-06T00:26:52.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.922394 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:26:52.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.922426 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:26:52.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:52.922434 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:26:53.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.036674 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-06T00:26:53.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.339954 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:26:53.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.339985 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:26:53.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.339994 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-06T00:26:53.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.420689 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:26:53.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.420713 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:26:53.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.420719 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:26:53.673 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' smithi066-lv-4 = '' ']' 2022-03-06T00:26:53.674 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get pv smithi066-lv-4 -o 'jsonpath={.spec.local.path}' 2022-03-06T00:26:53.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.890538 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:26:53.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.890571 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:26:53.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:53.890590 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:26:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:+ target_path=/dev/vg_nvme/lv_4 2022-03-06T00:26:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:++ echo smithi066-lv-4 2022-03-06T00:26:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f1 -d - 2022-03-06T00:26:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:+ host=smithi066 2022-03-06T00:26:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch device zap smithi066 /dev/vg_nvme/lv_4 --force 2022-03-06T00:26:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi066 /dev/vg_nvme/lv_4 --force 2022-03-06T00:26:54.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.148506 I | ceph-csi: successfully started CSI CephFS driver 2022-03-06T00:26:54.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.215392 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:26:54.227 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:26:54.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.252670 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:26:54.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.252898 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:26:54.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.449719 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:26:54.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.479976 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-06T00:26:54.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.479996 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:26:54.491 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.485763 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-06T00:26:54.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.731690 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:26:54.887 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_4 on smithi066 zapped 2022-03-06T00:26:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:+ zap_completion=0 2022-03-06T00:26:54.907 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 0 = 0 ']' 2022-03-06T00:26:54.907 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-06T00:26:54.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:54.954830 I | op-mon: start running mons 2022-03-06T00:26:55.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:55.043360 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:26:55.055 INFO:teuthology.orchestra.run.smithi007.stderr:+ zap_completion= 2022-03-06T00:26:55.055 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 3s 2022-03-06T00:26:55.542 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:55.536564 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789","10.109.72.196:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-06T00:26:55.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:55.716352 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:26:55.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:55.716611 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:26:57.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:57.065824 I | op-mon: targeting the mon count 3 2022-03-06T00:26:57.082 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:57.076359 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:26:57.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:57.734534 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:26:57.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:57.734556 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:26:58.056 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' '' = 0 ']' 2022-03-06T00:26:58.057 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl patch pv smithi066-lv-4 -p '{"spec":{"claimRef": null}}' 2022-03-06T00:26:58.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:58.080692 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:26:58.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:58.080721 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:26:58.127 INFO:teuthology.orchestra.run.smithi007.stdout:persistentvolume/smithi066-lv-4 patched 2022-03-06T00:26:58.130 INFO:teuthology.orchestra.run.smithi007.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-06T00:26:58.130 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-06T00:26:58.194 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:26:58.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:58.466327 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:26:58.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:58.466358 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:26:58.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:58.983905 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:26:58.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:58.983940 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:26:59.053 INFO:teuthology.orchestra.run.smithi007.stdout:['Success'] 2022-03-06T00:26:59.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.050288 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-06T00:26:59.057 INFO:tasks.rook.operator.smithi007.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", AllowUnsupported: true}, 2022-03-06T00:26:59.057 INFO:tasks.rook.operator.smithi007.stdout: Storage: v1.StorageScopeSpec{ 2022-03-06T00:26:59.058 INFO:tasks.rook.operator.smithi007.stdout: ... // 3 identical fields 2022-03-06T00:26:59.058 INFO:tasks.rook.operator.smithi007.stdout: Config: nil, 2022-03-06T00:26:59.058 INFO:tasks.rook.operator.smithi007.stdout: Selection: {}, 2022-03-06T00:26:59.059 INFO:tasks.rook.operator.smithi007.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-06T00:26:59.059 INFO:tasks.rook.operator.smithi007.stdout: ... // 9 identical elements 2022-03-06T00:26:59.060 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi066-lv-2", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi066-lv-2", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:26:59.062 INFO:tasks.rook.operator.smithi007.stdout: {Name: "smithi066-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi066-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-06T00:26:59.063 INFO:tasks.rook.operator.smithi007.stdout:+ { 2022-03-06T00:26:59.063 INFO:tasks.rook.operator.smithi007.stdout:+ Name: "smithi066-lv-4", 2022-03-06T00:26:59.064 INFO:tasks.rook.operator.smithi007.stdout:+ Count: 1, 2022-03-06T00:26:59.064 INFO:tasks.rook.operator.smithi007.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-06T00:26:59.064 INFO:tasks.rook.operator.smithi007.stdout:+ }, 2022-03-06T00:26:59.065 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-06T00:26:59.065 INFO:tasks.rook.operator.smithi007.stdout: }, 2022-03-06T00:26:59.065 INFO:tasks.rook.operator.smithi007.stdout: Annotations: nil, 2022-03-06T00:26:59.066 INFO:tasks.rook.operator.smithi007.stdout: Labels: nil, 2022-03-06T00:26:59.066 INFO:tasks.rook.operator.smithi007.stdout: ... // 20 identical fields 2022-03-06T00:26:59.066 INFO:tasks.rook.operator.smithi007.stdout: } 2022-03-06T00:26:59.066 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.050415 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-06T00:26:59.067 INFO:tasks.rook.operator.smithi007.stdout:I0306 00:26:59.050583 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-06T00:26:59.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.065494 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:26:59.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.065516 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:26:59.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.065541 I | operator: setting up schemes 2022-03-06T00:26:59.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.068950 I | operator: setting up the controller-runtime manager 2022-03-06T00:26:59.084 INFO:teuthology.orchestra.run.smithi007.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-06T00:26:59.155 INFO:teuthology.orchestra.run.smithi007.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-06T00:26:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=0 2022-03-06T00:26:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 0 '!=' 12 ']' 2022-03-06T00:26:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-06T00:26:59.158 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-06T00:26:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-06T00:26:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:26:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-06T00:26:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:26:59.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.341394 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:26:59.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.341428 I | op-mon: checking for basic quorum with existing mons 2022-03-06T00:26:59.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.373419 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:26:59.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404236 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:26:59.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404268 I | ceph-cluster-controller: successfully started 2022-03-06T00:26:59.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404308 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:26:59.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404322 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:26:59.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404335 I | ceph-block-pool-controller: successfully started 2022-03-06T00:26:59.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404347 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:26:59.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404360 I | ceph-object-realm-controller: successfully started 2022-03-06T00:26:59.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404370 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:26:59.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404378 I | ceph-object-zone-controller: successfully started 2022-03-06T00:26:59.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404492 I | ceph-object-controller: successfully started 2022-03-06T00:26:59.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404515 I | ceph-file-controller: successfully started 2022-03-06T00:26:59.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404535 I | ceph-nfs-controller: successfully started 2022-03-06T00:26:59.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404551 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:26:59.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404567 I | ceph-client-controller: successfully started 2022-03-06T00:26:59.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404579 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:26:59.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404595 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:26:59.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404607 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:26:59.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404642 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:26:59.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404656 I | ceph-bucket-topic: successfully started 2022-03-06T00:26:59.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404674 I | ceph-bucket-notification: successfully started 2022-03-06T00:26:59.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404701 I | ceph-bucket-notification: successfully started 2022-03-06T00:26:59.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.404716 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:26:59.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.405641 I | operator: starting the controller-runtime manager 2022-03-06T00:26:59.419 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.411539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-06T00:26:59.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.411763 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:26:59.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.440443 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:26:59.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.460651 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Detecting Ceph version LastHeartbeatTime:2022-03-06 00:26:59.453929603 +0000 UTC m=+442.645543750 LastTransitionTime:2022-03-06 00:26:59.453929315 +0000 UTC m=+442.645543501}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:26:59.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.460673 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:26:59.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.511004 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:26:59.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.511073 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:26:59.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.511092 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:26:59.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.511443 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:26:59.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.546528 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:26:59.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.557590 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:26:59.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.557649 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-06T00:26:59.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.557663 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:26:59.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.557720 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:26:59.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.559495 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-06T00:26:59.580 INFO:tasks.rook.operator.smithi007.stdout:I0306 00:26:59.559582 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-06T00:26:59.596 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.590436 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:26:59.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.697824 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-06T00:26:59.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.781464 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-06T00:26:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-06T00:26:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-06T00:26:59.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:26:59.887848 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-06T00:27:00.205 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198470 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-06T00:27:00.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198502 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-06T00:27:00.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198511 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-06T00:27:00.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198518 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-06T00:27:00.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198526 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-06T00:27:00.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198534 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-06T00:27:00.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198543 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-06T00:27:00.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198551 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-06T00:27:00.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198560 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-06T00:27:00.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198569 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-06T00:27:00.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198578 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-06T00:27:00.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198587 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-06T00:27:00.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198595 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-06T00:27:00.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198602 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-06T00:27:00.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198609 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-06T00:27:00.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198617 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-06T00:27:00.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198686 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:27:00.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:00.198703 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:27:03.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:03.116927 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-06T00:27:06.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:06.282496 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-06T00:27:07.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.446992 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-06T00:27:08.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447029 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-06T00:27:08.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447041 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:27:08.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447047 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-06T00:27:08.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447054 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:27:08.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447061 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-06T00:27:08.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447068 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:27:08.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447074 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-06T00:27:08.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447081 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:27:08.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447087 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-06T00:27:08.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447094 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:27:08.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447100 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-06T00:27:08.610 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447107 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:27:08.610 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447114 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-06T00:27:08.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447121 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-06T00:27:08.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447129 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-06T00:27:08.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447138 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-06T00:27:08.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447145 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-06T00:27:08.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447153 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-06T00:27:08.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447159 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:27:08.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447165 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-06T00:27:08.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447173 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-06T00:27:08.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447179 I | ceph-csi: Kubernetes version is 1.23 2022-03-06T00:27:08.615 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447188 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-06T00:27:08.615 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.447194 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-06T00:27:08.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.457644 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-06T00:27:08.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.463762 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:27:08.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.463783 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:27:08.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.463787 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:27:08.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.463791 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:27:08.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.463794 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:27:08.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.463797 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:27:08.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.463801 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-06T00:27:08.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.662750 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:27:08.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.662782 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:27:08.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.662794 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:27:08.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:07.885767 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-06T00:27:08.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.058422 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-06T00:27:08.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.058457 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-06T00:27:08.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.058471 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-06T00:27:08.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.117699 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-06T00:27:08.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.117733 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-06T00:27:08.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.117746 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-06T00:27:08.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.166242 I | ceph-csi: successfully started CSI CephFS driver 2022-03-06T00:27:08.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.367960 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:27:08.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.397694 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-06T00:27:08.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.397722 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-06T00:27:08.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:08.402563 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-06T00:27:09.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:09.287575 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-06T00:27:17.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:17.872343 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:27:17.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:17.872375 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:27:17.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:17.872389 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:27:17.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:17.873774 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:27:17.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:17.873799 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-06T00:27:17.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:17.873817 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-10974 quincy", pursuing 2022-03-06T00:27:18.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:18.172835 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:27:18.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:18.361083 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:27:18.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:18.377741 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:27:18.385 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:18.377894 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:27:18.385 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:18.377958 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:27:18.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:18.378340 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:27:19.070 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:19.063163 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:27:19.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:19.064540 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-10974 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97" 2022-03-06T00:27:19.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:19.723274 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-06 00:27:19.086475524 +0000 UTC m=+462.278089638 LastTransitionTime:2022-03-06 00:27:19.086475315 +0000 UTC m=+462.278089438}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:27:19.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:19.896699 I | op-mon: start running mons 2022-03-06T00:27:19.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:19.978849 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:27:20.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:20.089994 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789","10.109.72.196:6789"]}] data:b=10.98.23.111:6789,c=10.109.72.196:6789,a=10.102.190.159:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-06T00:27:20.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:20.107949 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:27:20.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:20.108165 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:27:21.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:21.530459 I | op-mon: targeting the mon count 3 2022-03-06T00:27:21.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:21.539027 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:27:21.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:21.897736 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:27:21.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:21.897761 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:27:22.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:22.257895 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:27:22.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:22.257924 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:27:22.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:22.626125 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:27:22.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:22.626159 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:27:22.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:22.976621 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:27:22.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:22.976642 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:27:23.327 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:23.321065 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:27:23.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:23.321099 I | op-mon: checking for basic quorum with existing mons 2022-03-06T00:27:23.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:23.467578 I | op-mon: mon "a" endpoint is [v2:10.102.190.159:3300,v1:10.102.190.159:6789] 2022-03-06T00:27:23.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:23.632758 I | op-mon: mon "b" endpoint is [v2:10.98.23.111:3300,v1:10.98.23.111:6789] 2022-03-06T00:27:23.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:23.930987 I | op-mon: mon "c" endpoint is [v2:10.109.72.196:3300,v1:10.109.72.196:6789] 2022-03-06T00:27:24.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:24.530261 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789","10.109.72.196:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-06T00:27:25.135 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:25.128254 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:27:25.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:25.128531 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:27:25.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:25.554117 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:27:25.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:25.569275 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:27:25.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:25.569309 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:26.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:26.141970 I | op-mon: mons running: [a b c] 2022-03-06T00:27:26.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:26.572293 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:26.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:26.581688 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-06T00:27:26.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:26.595380 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-06T00:27:26.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:26.595404 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:26.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:26.738331 I | op-mon: mons running: [a b c] 2022-03-06T00:27:27.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:27.154797 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:27.167 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:27.161815 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-06T00:27:27.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:27.171248 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-06T00:27:27.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:27.171271 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:27.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:27.339480 I | op-mon: mons running: [a b c] 2022-03-06T00:27:27.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:27.837992 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:27.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:27.838023 I | op-mon: mons created: 3 2022-03-06T00:27:28.252 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:28.245569 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:28.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:28.283261 I | op-mon: mons running: [a b c] 2022-03-06T00:27:28.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:28.720834 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:28.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:28.720906 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-06T00:27:28.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:28.720995 I | op-mon: start running mons 2022-03-06T00:27:28.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:28.727742 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:27:29.141 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:29.135132 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-06T00:27:29.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:29.329698 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789","10.109.72.196:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-06T00:27:29.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:29.564298 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-06T00:27:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-06T00:27:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-06T00:27:29.813 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-06T00:27:29.813 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-06T00:27:29.814 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:27:29.814 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-06T00:27:29.892 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:27:29.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:29.931188 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:27:29.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:29.931392 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:27:30.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:30.023752 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-06T00:27:30.440 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-06T00:27:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-06T00:27:32.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:32.530280 I | op-mon: targeting the mon count 3 2022-03-06T00:27:32.544 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:32.538481 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:27:32.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:32.933274 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-06T00:27:32.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:32.933303 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:27:33.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:33.130125 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-06T00:27:33.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:33.130153 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-06T00:27:33.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:33.297259 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-06T00:27:33.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:33.297296 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:27:33.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:33.558905 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-06T00:27:33.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:33.638090 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-06T00:27:33.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:33.638125 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:27:34.121 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.114170 I | cephclient: successfully enabled msgr2 protocol 2022-03-06T00:27:34.121 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.114204 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-06T00:27:34.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.114213 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-06T00:27:34.124 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.116211 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-06T00:27:34.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.116231 I | op-config: deleting "log file" option from the mon configuration database 2022-03-06T00:27:34.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.445856 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-06T00:27:34.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.445881 I | op-mon: checking for basic quorum with existing mons 2022-03-06T00:27:34.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.495690 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-06T00:27:34.598 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.592361 I | op-mgr: start running mgr 2022-03-06T00:27:34.599 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.592404 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-06T00:27:34.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:34.730695 I | op-mon: mon "a" endpoint is [v2:10.102.190.159:3300,v1:10.102.190.159:6789] 2022-03-06T00:27:35.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:35.929487 I | op-mon: mon "b" endpoint is [v2:10.98.23.111:3300,v1:10.98.23.111:6789] 2022-03-06T00:27:36.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:36.206777 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-06T00:27:36.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:36.217134 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-06T00:27:36.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:36.233142 I | op-mgr: setting services to point to mgr "a" 2022-03-06T00:27:37.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:37.531014 I | op-mon: mon "c" endpoint is [v2:10.109.72.196:3300,v1:10.109.72.196:6789] 2022-03-06T00:27:38.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.127934 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-06T00:27:38.945 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.128065 I | op-mgr: successful modules: balancer 2022-03-06T00:27:38.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.139979 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-06T00:27:38.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.140014 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-06T00:27:38.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.190244 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-06T00:27:38.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529569 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-1" 2022-03-06T00:27:38.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529610 I | op-osd: OSD PVC "smithi007-lv-1-data-07jksx" already exists 2022-03-06T00:27:38.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529623 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-2" 2022-03-06T00:27:38.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529636 I | op-osd: OSD PVC "smithi007-lv-2-data-0xh6k5" already exists 2022-03-06T00:27:38.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529644 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-3" 2022-03-06T00:27:38.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529656 I | op-osd: OSD PVC "smithi007-lv-3-data-059rmw" already exists 2022-03-06T00:27:38.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529664 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-4" 2022-03-06T00:27:38.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529674 I | op-osd: OSD PVC "smithi007-lv-4-data-0nknbb" already exists 2022-03-06T00:27:38.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529682 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi017-lv-1" 2022-03-06T00:27:38.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529695 I | op-osd: OSD PVC "smithi017-lv-1-data-0dfdbl" already exists 2022-03-06T00:27:38.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529706 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi017-lv-2" 2022-03-06T00:27:38.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529733 I | op-osd: OSD PVC "smithi017-lv-2-data-04dp76" already exists 2022-03-06T00:27:38.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529742 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi017-lv-3" 2022-03-06T00:27:38.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529755 I | op-osd: OSD PVC "smithi017-lv-3-data-0vkmbc" already exists 2022-03-06T00:27:38.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529766 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi017-lv-4" 2022-03-06T00:27:38.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529782 I | op-osd: OSD PVC "smithi017-lv-4-data-0ncm7h" already exists 2022-03-06T00:27:38.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529804 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-1" 2022-03-06T00:27:38.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529821 I | op-osd: OSD PVC "smithi066-lv-1-data-0r6fcr" already exists 2022-03-06T00:27:38.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529840 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-2" 2022-03-06T00:27:38.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529860 I | op-osd: OSD PVC "smithi066-lv-2-data-04bnc5" already exists 2022-03-06T00:27:38.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529876 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-3" 2022-03-06T00:27:38.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529892 I | op-osd: OSD PVC "smithi066-lv-3-data-0mh6gp" already exists 2022-03-06T00:27:38.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.529922 I | op-osd: creating 1 new PVCs for device set "smithi066-lv-4" 2022-03-06T00:27:38.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.730467 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.102.190.159:6789","10.98.23.111:6789","10.109.72.196:6789"]}] data:a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 mapping:{"node":{"a":{"Name":"smithi066","Hostname":"smithi066","Address":"172.21.15.66"},"b":{"Name":"smithi017","Hostname":"smithi017","Address":"172.21.15.17"},"c":{"Name":"smithi007","Hostname":"smithi007","Address":"172.21.15.7"}}} maxMonId:2] 2022-03-06T00:27:38.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.951350 I | op-osd: successfully provisioned PVC "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:27:38.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988192 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-07jksx" 2022-03-06T00:27:38.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988219 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-1-data-07jksx" because OSD daemon using the PVC already exists 2022-03-06T00:27:38.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988224 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:27:38.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988227 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-2-data-0xh6k5" because OSD daemon using the PVC already exists 2022-03-06T00:27:38.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988231 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-059rmw" 2022-03-06T00:27:38.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988236 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-3-data-059rmw" because OSD daemon using the PVC already exists 2022-03-06T00:27:38.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988239 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0nknbb" 2022-03-06T00:27:38.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988243 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-4-data-0nknbb" because OSD daemon using the PVC already exists 2022-03-06T00:27:38.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988246 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:27:38.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988250 I | op-osd: skipping OSD prepare job creation for PVC "smithi017-lv-1-data-0dfdbl" because OSD daemon using the PVC already exists 2022-03-06T00:27:38.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988254 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-2-data-04dp76" 2022-03-06T00:27:38.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988258 I | op-osd: skipping OSD prepare job creation for PVC "smithi017-lv-2-data-04dp76" because OSD daemon using the PVC already exists 2022-03-06T00:27:38.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988261 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:27:38.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988265 I | op-osd: skipping OSD prepare job creation for PVC "smithi017-lv-3-data-0vkmbc" because OSD daemon using the PVC already exists 2022-03-06T00:27:39.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988269 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:27:39.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988273 I | op-osd: skipping OSD prepare job creation for PVC "smithi017-lv-4-data-0ncm7h" because OSD daemon using the PVC already exists 2022-03-06T00:27:39.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988277 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:27:39.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988280 I | op-osd: skipping OSD prepare job creation for PVC "smithi066-lv-1-data-0r6fcr" because OSD daemon using the PVC already exists 2022-03-06T00:27:39.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988285 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-04bnc5" 2022-03-06T00:27:39.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988292 I | op-osd: skipping OSD prepare job creation for PVC "smithi066-lv-2-data-04bnc5" because OSD daemon using the PVC already exists 2022-03-06T00:27:39.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988300 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:27:39.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988308 I | op-osd: skipping OSD prepare job creation for PVC "smithi066-lv-3-data-0mh6gp" because OSD daemon using the PVC already exists 2022-03-06T00:27:39.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:38.988315 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:27:39.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:39.606674 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-06T00:27:39.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:39.666626 I | op-mgr: successful modules: dashboard 2022-03-06T00:27:39.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:39.671968 I | op-mgr: successful modules: prometheus 2022-03-06T00:27:39.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:39.766237 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:27:39.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:39.766276 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-06T00:27:39.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:39.766286 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-06T00:27:39.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:39.980835 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-06T00:27:39.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:39.981165 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-06T00:27:40.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.128293 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bwwkw is "starting" 2022-03-06T00:27:40.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.230319 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:40.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.236478 I | op-osd: updating OSD 1 on PVC "smithi007-lv-1-data-07jksx" 2022-03-06T00:27:40.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.236511 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-07jksx" 2022-03-06T00:27:40.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.405503 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:40.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.438266 I | op-osd: updating OSD 10 on PVC "smithi066-lv-2-data-04bnc5" 2022-03-06T00:27:40.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.438302 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-04bnc5" 2022-03-06T00:27:40.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.561990 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-06T00:27:40.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.571261 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-06T00:27:40.577 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.571290 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:40.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.680861 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:40.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.700269 I | op-osd: updating OSD 11 on PVC "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:27:40.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:40.700306 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:27:41.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:41.065127 I | op-mgr: successful modules: orchestrator modules 2022-03-06T00:27:41.145 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:41.138939 I | op-mon: mons running: [a b c] 2022-03-06T00:27:41.349 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:41.343982 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:41.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:41.441764 I | op-osd: updating OSD 2 on PVC "smithi007-lv-3-data-059rmw" 2022-03-06T00:27:41.449 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:41.441821 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-059rmw" 2022-03-06T00:27:41.588 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:41.573252 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:41.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:41.837179 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-06T00:27:42.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:42.244547 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-06T00:27:42.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:42.244580 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:42.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:42.282671 I | op-mon: mons running: [a b c] 2022-03-06T00:27:42.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:42.543771 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:42.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:42.638387 I | op-osd: updating OSD 3 on PVC "smithi007-lv-4-data-0nknbb" 2022-03-06T00:27:42.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:42.638424 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0nknbb" 2022-03-06T00:27:42.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:42.703195 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:43.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.035687 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-06T00:27:43.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.444299 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-06T00:27:43.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.444339 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:43.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.476536 I | op-mon: mons running: [a b c] 2022-03-06T00:27:43.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.745596 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:43.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.838091 I | op-osd: updating OSD 4 on PVC "smithi017-lv-2-data-04dp76" 2022-03-06T00:27:43.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.838119 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-2-data-04dp76" 2022-03-06T00:27:43.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.902836 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:43.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:43.902867 I | op-mon: mons created: 3 2022-03-06T00:27:44.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:44.317111 I | op-mon: waiting for mon quorum with [a b c] 2022-03-06T00:27:44.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:44.358372 I | op-mon: mons running: [a b c] 2022-03-06T00:27:44.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:44.570000 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:44.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:44.638833 I | op-osd: updating OSD 5 on PVC "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:27:44.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:44.638864 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:27:44.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:44.776357 I | op-mon: Monitors in quorum: [a b c] 2022-03-06T00:27:44.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:44.776406 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-06T00:27:45.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:45.203340 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-06T00:27:45.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:45.344860 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:45.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:45.492788 I | op-osd: updating OSD 6 on PVC "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:27:45.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:45.492815 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:27:45.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:45.697719 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-06T00:27:46.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:46.118606 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-06T00:27:46.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:46.144986 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:46.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:46.237090 I | op-osd: updating OSD 7 on PVC "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:27:46.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:46.237122 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:27:46.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:46.656900 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-06T00:27:46.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:46.656936 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-06T00:27:46.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:46.944779 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:47.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.038109 I | op-osd: updating OSD 8 on PVC "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:27:47.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.038143 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:27:47.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.098468 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-06T00:27:47.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.529154 I | cephclient: successfully enabled msgr2 protocol 2022-03-06T00:27:47.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.529196 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-06T00:27:47.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.529205 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-06T00:27:47.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.742477 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:47.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.838704 I | op-osd: updating OSD 9 on PVC "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:27:47.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.838746 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:27:47.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.952954 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-06T00:27:47.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.988750 I | op-mgr: start running mgr 2022-03-06T00:27:47.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:47.988783 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-06T00:27:48.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:48.689525 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-06T00:27:48.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:48.844214 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-06T00:27:49.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.036091 I | op-mgr: setting services to point to mgr "a" 2022-03-06T00:27:49.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.117915 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-06T00:27:49.124 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.118017 I | op-mgr: successful modules: balancer 2022-03-06T00:27:49.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.172066 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-06T00:27:49.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.172105 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-06T00:27:49.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.273502 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-06T00:27:49.308 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299090 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-1" 2022-03-06T00:27:49.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299135 I | op-osd: OSD PVC "smithi007-lv-1-data-07jksx" already exists 2022-03-06T00:27:49.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299149 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-2" 2022-03-06T00:27:49.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299160 I | op-osd: OSD PVC "smithi007-lv-2-data-0xh6k5" already exists 2022-03-06T00:27:49.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299168 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-3" 2022-03-06T00:27:49.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299179 I | op-osd: OSD PVC "smithi007-lv-3-data-059rmw" already exists 2022-03-06T00:27:49.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299193 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi007-lv-4" 2022-03-06T00:27:49.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299207 I | op-osd: OSD PVC "smithi007-lv-4-data-0nknbb" already exists 2022-03-06T00:27:49.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299216 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi017-lv-1" 2022-03-06T00:27:49.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299227 I | op-osd: OSD PVC "smithi017-lv-1-data-0dfdbl" already exists 2022-03-06T00:27:49.314 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299235 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi017-lv-2" 2022-03-06T00:27:49.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299250 I | op-osd: OSD PVC "smithi017-lv-2-data-04dp76" already exists 2022-03-06T00:27:49.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299267 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi017-lv-3" 2022-03-06T00:27:49.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299287 I | op-osd: OSD PVC "smithi017-lv-3-data-0vkmbc" already exists 2022-03-06T00:27:49.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299301 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi017-lv-4" 2022-03-06T00:27:49.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299321 I | op-osd: OSD PVC "smithi017-lv-4-data-0ncm7h" already exists 2022-03-06T00:27:49.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299335 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-1" 2022-03-06T00:27:49.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299348 I | op-osd: OSD PVC "smithi066-lv-1-data-0r6fcr" already exists 2022-03-06T00:27:49.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299363 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-2" 2022-03-06T00:27:49.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299383 I | op-osd: OSD PVC "smithi066-lv-2-data-04bnc5" already exists 2022-03-06T00:27:49.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299395 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-3" 2022-03-06T00:27:49.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299414 I | op-osd: OSD PVC "smithi066-lv-3-data-0mh6gp" already exists 2022-03-06T00:27:49.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299429 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi066-lv-4" 2022-03-06T00:27:49.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.299448 I | op-osd: OSD PVC "smithi066-lv-4-data-0bwwkw" already exists 2022-03-06T00:27:49.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460275 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-07jksx" 2022-03-06T00:27:49.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460304 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-1-data-07jksx" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460311 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:27:49.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460316 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-2-data-0xh6k5" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460321 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-059rmw" 2022-03-06T00:27:49.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460325 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-3-data-059rmw" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460330 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0nknbb" 2022-03-06T00:27:49.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460335 I | op-osd: skipping OSD prepare job creation for PVC "smithi007-lv-4-data-0nknbb" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460341 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:27:49.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460345 I | op-osd: skipping OSD prepare job creation for PVC "smithi017-lv-1-data-0dfdbl" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460350 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-2-data-04dp76" 2022-03-06T00:27:49.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460355 I | op-osd: skipping OSD prepare job creation for PVC "smithi017-lv-2-data-04dp76" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460359 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:27:49.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460364 I | op-osd: skipping OSD prepare job creation for PVC "smithi017-lv-3-data-0vkmbc" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460370 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:27:49.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460375 I | op-osd: skipping OSD prepare job creation for PVC "smithi017-lv-4-data-0ncm7h" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460380 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:27:49.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460386 I | op-osd: skipping OSD prepare job creation for PVC "smithi066-lv-1-data-0r6fcr" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460391 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-04bnc5" 2022-03-06T00:27:49.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460398 I | op-osd: skipping OSD prepare job creation for PVC "smithi066-lv-2-data-04bnc5" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460403 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:27:49.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460410 I | op-osd: skipping OSD prepare job creation for PVC "smithi066-lv-3-data-0mh6gp" because OSD daemon using the PVC already exists 2022-03-06T00:27:49.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.460419 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:27:49.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.471968 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi066-lv-4-data-0bwwkw. Status={Conditions:[] StartTime:2022-03-06 00:27:39 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-06T00:27:49.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.471991 I | op-osd: started OSD provisioning job for PVC "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:27:49.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.472002 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-06T00:27:49.478 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.472009 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-06T00:27:49.480 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.474398 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bwwkw is "starting" 2022-03-06T00:27:49.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.576155 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:49.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.638439 I | op-osd: updating OSD 1 on PVC "smithi007-lv-1-data-07jksx" 2022-03-06T00:27:49.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:49.638472 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-1-data-07jksx" 2022-03-06T00:27:50.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:50.076417 I | op-mgr: successful modules: dashboard 2022-03-06T00:27:50.082 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:50.076514 I | op-mgr: successful modules: prometheus 2022-03-06T00:27:50.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:50.080216 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-06T00:27:50.366 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:50.360275 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:50.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:50.435267 I | op-osd: updating OSD 10 on PVC "smithi066-lv-2-data-04bnc5" 2022-03-06T00:27:50.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:50.435298 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-2-data-04bnc5" 2022-03-06T00:27:51.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:51.191447 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:51.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:51.237396 I | op-osd: updating OSD 11 on PVC "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:27:51.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:51.237432 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-3-data-0mh6gp" 2022-03-06T00:27:51.950 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:51.944565 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:52.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:52.038000 I | op-osd: updating OSD 2 on PVC "smithi007-lv-3-data-059rmw" 2022-03-06T00:27:52.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:52.038074 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-3-data-059rmw" 2022-03-06T00:27:52.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:52.267197 I | op-mgr: successful modules: orchestrator modules 2022-03-06T00:27:52.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:52.744049 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:52.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:52.838525 I | op-osd: updating OSD 3 on PVC "smithi007-lv-4-data-0nknbb" 2022-03-06T00:27:52.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:52.838562 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-4-data-0nknbb" 2022-03-06T00:27:53.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:53.567717 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:53.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:53.659920 I | op-osd: updating OSD 4 on PVC "smithi017-lv-2-data-04dp76" 2022-03-06T00:27:53.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:53.659959 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-2-data-04dp76" 2022-03-06T00:27:54.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:54.372673 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:54.443 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:54.438122 I | op-osd: updating OSD 5 on PVC "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:27:54.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:54.438153 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-4-data-0ncm7h" 2022-03-06T00:27:55.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:55.153114 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:55.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:55.243513 I | op-osd: updating OSD 6 on PVC "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:27:55.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:55.243554 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-3-data-0vkmbc" 2022-03-06T00:27:55.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:55.941789 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:56.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:56.041287 I | op-osd: updating OSD 7 on PVC "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:27:56.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:56.041325 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-1-data-0r6fcr" 2022-03-06T00:27:56.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:56.743086 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:56.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:56.838067 I | op-osd: updating OSD 8 on PVC "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:27:56.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:56.838092 I | op-osd: OSD will have its main bluestore block on "smithi017-lv-1-data-0dfdbl" 2022-03-06T00:27:57.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:57.543546 I | op-osd: skipping osd checks for ok-to-stop 2022-03-06T00:27:57.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:57.637851 I | op-osd: updating OSD 9 on PVC "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:27:57.643 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:27:57.637883 I | op-osd: OSD will have its main bluestore block on "smithi007-lv-2-data-0xh6k5" 2022-03-06T00:28:00.441 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-06T00:28:00.442 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-06T00:28:00.443 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-06T00:28:00.443 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-06T00:28:00.444 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:28:00.444 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-06T00:28:00.552 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:28:00.862 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:00.856890 I | op-osd: OSD orchestration status for node smithi066-lv-4-data-0bwwkw is "orchestrating" 2022-03-06T00:28:00.862 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:00.857023 I | op-osd: OSD orchestration status for node smithi066-lv-4-data-0bwwkw is "orchestrating" 2022-03-06T00:28:01.052 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=11 2022-03-06T00:28:01.053 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-06T00:28:01.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:01.242431 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bwwkw is "orchestrating" 2022-03-06T00:28:01.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:01.242473 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bwwkw is "orchestrating" 2022-03-06T00:28:07.366 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.359611 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bwwkw is "completed" 2022-03-06T00:28:07.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.359641 I | op-osd: creating OSD 0 on PVC "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:28:07.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.359652 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:28:07.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.359704 I | op-osd: OSD orchestration status for PVC smithi066-lv-4-data-0bwwkw is "completed" 2022-03-06T00:28:07.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.359723 I | op-osd: creating OSD 0 on PVC "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:28:07.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.359735 I | op-osd: OSD will have its main bluestore block on "smithi066-lv-4-data-0bwwkw" 2022-03-06T00:28:07.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.486397 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Processing OSD 0 on PVC "smithi066-lv-4-data-0bwwkw" LastHeartbeatTime:2022-03-06 00:28:07.445523557 +0000 UTC m=+510.637137650 LastTransitionTime:2022-03-06 00:28:07.445523403 +0000 UTC m=+510.637137516}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:28:07.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.607084 E | op-osd: failed to create OSD 0 on PVC "smithi066-lv-4-data-0bwwkw": failed to create deployment for OSD 0 on PVC "smithi066-lv-4-data-0bwwkw": deployments.apps "rook-ceph-osd-0" already exists 2022-03-06T00:28:07.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:07.626596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-06T00:28:07.632 INFO:tasks.rook.operator.smithi007.stdout:failed to create OSD 0 on PVC "smithi066-lv-4-data-0bwwkw": failed to create deployment for OSD 0 on PVC "smithi066-lv-4-data-0bwwkw": deployments.apps "rook-ceph-osd-0" already exists 2022-03-06T00:28:08.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:08.050073 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-06T00:28:08.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:28:08.050104 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-06T00:28:31.054 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 11 '!=' 12 ']' 2022-03-06T00:28:31.055 INFO:teuthology.orchestra.run.smithi007.stderr:+ echo 'waiting for osd to come back up' 2022-03-06T00:28:31.056 INFO:teuthology.orchestra.run.smithi007.stdout:waiting for osd to come back up 2022-03-06T00:28:31.056 INFO:teuthology.orchestra.run.smithi007.stderr:++ toolbox ceph osd stat 2022-03-06T00:28:31.057 INFO:teuthology.orchestra.run.smithi007.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-06T00:28:31.057 INFO:teuthology.orchestra.run.smithi007.stderr:++ cut -f3 -d ' ' 2022-03-06T00:28:31.124 INFO:teuthology.orchestra.run.smithi007.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-06T00:28:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:+ num_osd=12 2022-03-06T00:28:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:+ sleep 30s 2022-03-06T00:29:01.614 INFO:teuthology.orchestra.run.smithi007.stderr:+ '[' 12 '!=' 12 ']' 2022-03-06T00:29:01.619 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-06T00:29:01.633 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch status 2022-03-06T00:29:02.350 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: rook 2022-03-06T00:29:02.350 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2022-03-06T00:29:02.376 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ps 2022-03-06T00:29:02.941 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-06T00:29:02.941 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi007 running (5m) 0s ago 5m - - 02136b44099a 2022-03-06T00:29:02.941 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi017 running (4m) 0s ago 4m - - 02136b44099a 2022-03-06T00:29:02.941 INFO:teuthology.orchestra.run.smithi007.stdout:crashcollector.crash smithi066 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.942 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.a smithi066 running (5m) 0s ago 5m - - 02136b44099a 2022-03-06T00:29:02.942 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi066 running (7m) 0s ago 7m - - 02136b44099a 2022-03-06T00:29:02.942 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi017 running (7m) 0s ago 7m - - 02136b44099a 2022-03-06T00:29:02.942 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (6m) 0s ago 6m - - 02136b44099a 2022-03-06T00:29:02.943 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi066 running (55s) 0s ago 55s - - 02136b44099a 2022-03-06T00:29:02.943 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.943 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 smithi066 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.943 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 smithi066 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.944 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.944 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.944 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi017 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.944 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi017 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.944 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi017 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi066 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 smithi017 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.945 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 smithi007 running (3m) 0s ago 3m - - 02136b44099a 2022-03-06T00:29:02.966 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls 2022-03-06T00:29:03.694 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-06T00:29:03.695 INFO:teuthology.orchestra.run.smithi007.stdout:crash 3/3 0s ago 5m * 2022-03-06T00:29:03.695 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 1/1 0s ago 5m count:1 2022-03-06T00:29:03.695 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 0s ago 7m count:3 2022-03-06T00:29:03.696 INFO:teuthology.orchestra.run.smithi007.stdout:osd 24 0s ago 3m 2022-03-06T00:29:03.696 INFO:teuthology.orchestra.run.smithi007.stdout:osd.all-available-devices 0 0s ago - * 2022-03-06T00:29:03.722 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch host ls 2022-03-06T00:29:04.294 INFO:teuthology.orchestra.run.smithi007.stdout:HOST ADDR LABELS STATUS 2022-03-06T00:29:04.295 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 172.21.15.7/smithi007 2022-03-06T00:29:04.295 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 172.21.15.17/smithi017 2022-03-06T00:29:04.295 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 172.21.15.66/smithi066 2022-03-06T00:29:04.295 INFO:teuthology.orchestra.run.smithi007.stdout:3 hosts in cluster 2022-03-06T00:29:04.314 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch device ls 2022-03-06T00:29:04.974 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-06T00:29:04.974 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:29:04.975 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:29:04.975 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:29:04.975 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:29:04.975 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:29:04.976 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:29:04.976 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:29:04.976 INFO:teuthology.orchestra.run.smithi007.stdout:smithi017 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:29:04.976 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-06T00:29:04.976 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-06T00:29:04.977 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-06T00:29:04.977 INFO:teuthology.orchestra.run.smithi007.stdout:smithi066 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-06T00:29:04.999 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch apply rgw foo 2022-03-06T00:29:05.588 INFO:teuthology.orchestra.run.smithi007.stdout:Created 2022-03-06T00:29:05.589 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:05.582462 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-06T00:29:05.607 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch apply mds foo 2022-03-06T00:29:05.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:05.673196 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:05.676 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:05.673276 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:29:06.183 INFO:teuthology.orchestra.run.smithi007.stdout:Created 2022-03-06T00:29:06.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:06.175906 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-06T00:29:06.565 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch apply rbd-mirror 2022-03-06T00:29:06.566 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:06.354450 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:29:06.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:06.370178 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:06.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:06.370222 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:29:07.330 INFO:teuthology.orchestra.run.smithi007.stdout:Success 2022-03-06T00:29:07.349 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch apply nfs foo --port 12777 2022-03-06T00:29:07.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:07.362910 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:07.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:07.362949 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97... 2022-03-06T00:29:12.538 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-06T00:29:12.538 INFO:teuthology.orchestra.run.smithi007.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-06T00:29:12.563 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-06T00:29:12.574 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:29:12.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:12.937470 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:29:13.261 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:29:13.262 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:29:13.093488Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:13.093488Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:29:13.093488Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:29:13.093488Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:29:13.093488Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:29:13.093488Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-06T00:29:13.093488Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:13.093488Z", "running": 0, "size": 1}}] 2022-03-06T00:29:13.284 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch rm mds.foo 2022-03-06T00:29:13.364 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:13.361067 W | ceph-rbd-mirror-controller: (DEPRECATED) use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. 2022-03-06T00:29:13.364 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:13.361097 I | ceph-rbd-mirror-controller: configure rbd-mirroring with 1 workers 2022-03-06T00:29:13.364 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:13.361113 I | cephclient: getting or creating ceph auth key "client.rbd-mirror.a" 2022-03-06T00:29:13.993 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephfilesystems/foo 2022-03-06T00:29:14.012 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch rm rbd-mirror 2022-03-06T00:29:14.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:14.304472 I | ceph-rbd-mirror-controller: "rook-ceph-rbd-mirror-a" deployment started 2022-03-06T00:29:14.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:14.589924 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:29:14.676 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-06T00:29:14.699 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch rm rgw.foo 2022-03-06T00:29:14.765 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:14.763262 I | ceph-spec: object "rook-ceph-rbd-mirror-a" matched on delete, reconciling 2022-03-06T00:29:15.283 INFO:teuthology.orchestra.run.smithi007.stdout:Removed cephobjectstores/foo 2022-03-06T00:29:15.308 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:29:15.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:15.721846 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:29:15.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:15.751665 I | ceph-spec: detected ceph image version: "17.0.0-10974 quincy" 2022-03-06T00:29:16.187 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:29:16.187 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:29:15.867707Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:15.867707Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:29:15.867707Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:29:15.867707Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:29:15.867707Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:29:15.867707Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:15.867707Z", "running": 0, "size": 1}}] 2022-03-06T00:29:17.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:17.299198 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-06T00:29:17.392 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:17.389632 I | ceph-object-controller: reconciling object store deployments 2022-03-06T00:29:17.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:17.471707 I | ceph-object-controller: ceph object store gateway service running at 10.107.55.160 2022-03-06T00:29:17.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:17.471756 I | ceph-object-controller: reconciling object store pools 2022-03-06T00:29:17.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:17.715935 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-06T00:29:17.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:17.792132 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-06T00:29:19.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:19.261345 I | op-mds: setting mds config flags 2022-03-06T00:29:19.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:19.261383 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-06T00:29:20.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:20.033442 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-06T00:29:20.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:20.047050 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-06T00:29:20.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:20.371061 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi017": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:29:22.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:22.623839 I | op-mds: setting mds config flags 2022-03-06T00:29:22.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:22.623893 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-06T00:29:24.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:24.421122 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:29:24.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:24.578986 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi066": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:29:26.208 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:29:26.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:26.690753 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-06T00:29:27.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:29:27.162 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:29:26.744646Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:26.744646Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:29:26.744646Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:29:26.744646Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:29:26.744646Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:29:26.744646Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:26.744646Z", "running": 0, "size": 1}}] 2022-03-06T00:29:29.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:29.485163 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-06T00:29:30.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:29.820495 I | ceph-file-controller: creating filesystem "foo" 2022-03-06T00:29:30.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:30.602324 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-06T00:29:30.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:30.606630 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-06T00:29:30.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:30.637091 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-06T00:29:30.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:30.699036 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-06T00:29:31.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:31.603739 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-06T00:29:31.634 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:31.632321 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:34.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:34.068290 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-06T00:29:37.183 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:29:37.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:37.309345 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:37.298+0000 7f00bbf36380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:37.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:37.298+0000 7f00bbf36380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:37.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:37.298+0000 7f00bbf36380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:37.312 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:37.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:37.331913 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:38.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:38.038345 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:38.030+0000 7fcb72e38380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:38.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:38.030+0000 7fcb72e38380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:38.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:38.030+0000 7fcb72e38380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:38.041 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:38.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:38.066537 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:38.109 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:29:38.109 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:29:37.842729Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:37.842729Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:29:37.842729Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:29:37.842729Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:29:37.842729Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:29:37.842729Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:37.842729Z", "running": 0, "size": 1}}] 2022-03-06T00:29:38.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:38.612346 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-06T00:29:38.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:38.721179 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:38.710+0000 7f05aa14f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:38.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:38.710+0000 7f05aa14f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:38.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:38.710+0000 7f05aa14f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:38.724 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:38.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:38.739748 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:39.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:39.336271 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:39.326+0000 7fa22db18380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:39.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:39.326+0000 7fa22db18380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:39.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:39.326+0000 7fa22db18380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:39.340 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:39.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:39.349644 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-06T00:29:39.360 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:39.358707 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:40.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:40.058062 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:40.046+0000 7f5b61e43380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:40.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:40.046+0000 7f5b61e43380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:40.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:40.046+0000 7f5b61e43380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:40.061 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:40.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:40.099250 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:40.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:40.856023 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:40.846+0000 7f2dc61f3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:40.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:40.846+0000 7f2dc61f3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:40.859 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:40.846+0000 7f2dc61f3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:40.859 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:40.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:40.895812 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:41.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:41.445630 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-06T00:29:41.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:41.445651 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-06T00:29:41.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:41.553038 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:41.542+0000 7fb7a5a18380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:41.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:41.542+0000 7fb7a5a18380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:41.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:41.542+0000 7fb7a5a18380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:41.556 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:41.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:41.581715 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:42.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:42.191035 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:42.178+0000 7fc752baa380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:42.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:42.178+0000 7fc752baa380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:42.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:42.178+0000 7fc752baa380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:42.195 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:42.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:42.206242 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:42.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:42.834839 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:42.822+0000 7fbd9ec4d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:42.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:42.822+0000 7fbd9ec4d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:42.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:42.822+0000 7fbd9ec4d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:42.838 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:43.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:43.232094 W | ceph-file-controller: failed to set filesystem "foo" status to "". failed to update object "rook-ceph/foo" status: Operation cannot be fulfilled on cephfilesystems.ceph.rook.io "foo": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:29:43.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:43.237419 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:43.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:43.240124 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-06T00:29:44.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:44.031011 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-06T00:29:44.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:44.492162 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-06T00:29:44.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:44.492193 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-06T00:29:44.495 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:44.492202 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-06T00:29:45.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:45.067440 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-06T00:29:45.407 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:45.405550 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:45.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:45.435942 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-06T00:29:46.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:46.015830 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-06T00:29:46.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:46.015852 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-06T00:29:46.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:46.015856 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-06T00:29:46.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:46.118892 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:46.106+0000 7f1de37b6380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:46.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:46.106+0000 7f1de37b6380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:46.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:46.106+0000 7f1de37b6380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:46.123 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:46.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:46.642722 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-06T00:29:46.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:46.642759 I | ceph-file-controller: downing filesystem "foo" 2022-03-06T00:29:48.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:48.011563 I | ceph-file-controller: downed filesystem "foo" 2022-03-06T00:29:48.142 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:29:51.268 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:51.266291 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:29:51.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:51.964829 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-06T00:29:51.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:51.964850 I | cephclient: purging pool "foo-metadata" (id=9) 2022-03-06T00:29:52.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:52.464329 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:29:52.434+0000 7f19dc43b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:29:52.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:52.434+0000 7f19dc43b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:29:52.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:29:52.434+0000 7f19dc43b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:29:52.467 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:29:52.746 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:29:54.246 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:29:52.533378Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:52.533378Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:29:52.533378Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:29:52.533378Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:29:52.533378Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:29:52.533378Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:29:52.533378Z", "running": 0, "size": 1}}] 2022-03-06T00:29:54.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:54.906367 I | cephclient: purge completed for pool "foo-metadata" 2022-03-06T00:29:55.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:55.313467 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-06T00:29:55.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:55.313491 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-06T00:29:57.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:57.560533 I | cephclient: purge completed for pool "foo-data0" 2022-03-06T00:29:57.566 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:57.560673 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-06T00:29:57.591 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:57.589937 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-06T00:29:57.594 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:57.590154 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-06T00:29:57.598 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:57.596517 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-06T00:29:57.598 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:29:57.596628 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-06T00:30:00.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:30:00.521954 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi017": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:30:00.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:30:00.769033 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi017": the object has been modified; please apply your changes to the latest version and try again 2022-03-06T00:30:02.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:30:02.748033 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:30:04.249 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:30:04.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:30:04.351760 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:30:04.338+0000 7f3f22b6d380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:30:04.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:30:04.338+0000 7f3f22b6d380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:30:04.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:30:04.338+0000 7f3f22b6d380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:30:04.355 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:30:04.984 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:30:04.985 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:30:04.800895Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:30:04.800895Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:30:04.800895Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:30:04.800895Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:30:04.800895Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:30:04.800895Z", "running": 0, "size": 1}}] 2022-03-06T00:30:15.020 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:30:15.824 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:30:15.824 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:30:15.593050Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:30:15.593050Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:30:15.593050Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:30:15.593050Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:30:15.593050Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:30:15.593050Z", "running": 0, "size": 1}}] 2022-03-06T00:30:24.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:30:24.841340 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:30:25.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:30:25.433736 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:30:25.425+0000 7efdc74b8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:30:25.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:30:25.425+0000 7efdc74b8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:30:25.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:30:25.425+0000 7efdc74b8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:30:25.438 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:30:25.846 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:30:26.623 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:30:26.623 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:30:26.427855Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:30:26.427855Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:30:26.427855Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:30:26.427855Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:30:26.427855Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:30:26.427855Z", "running": 0, "size": 1}}] 2022-03-06T00:30:36.652 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:30:37.499 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:30:37.500 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:30:37.261111Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:30:37.261111Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:30:37.261111Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:30:37.261111Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:30:37.261111Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:30:37.261111Z", "running": 0, "size": 1}}] 2022-03-06T00:30:47.541 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:30:48.310 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:30:48.311 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:30:48.094885Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:30:48.094885Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:30:48.094885Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:30:48.094885Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:30:48.094885Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:30:48.094885Z", "running": 0, "size": 1}}] 2022-03-06T00:30:58.339 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:30:59.107 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:30:59.107 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:30:58.895383Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:30:58.895383Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:30:58.895383Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:30:58.895383Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:30:58.895383Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:30:58.895383Z", "running": 0, "size": 1}}] 2022-03-06T00:31:06.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:31:06.418574 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:31:07.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:31:07.129878 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:31:07.120+0000 7f8399350380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:31:07.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:31:07.120+0000 7f8399350380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:31:07.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:31:07.120+0000 7f8399350380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:31:07.137 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:31:09.134 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:31:09.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:31:09.893 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:31:09.673310Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:31:09.673310Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:31:09.673310Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:31:09.673310Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:09.673310Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:09.673310Z", "running": 0, "size": 1}}] 2022-03-06T00:31:19.938 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:31:20.694 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:31:20.695 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:31:20.499372Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:31:20.499372Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:31:20.499372Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:31:20.499372Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:20.499372Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:20.499372Z", "running": 0, "size": 1}}] 2022-03-06T00:31:30.741 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:31:31.453 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:31:31.865 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:31:31.283473Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:31:31.283473Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:31:31.283473Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:31:31.283473Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:31.283473Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:31.283473Z", "running": 0, "size": 1}}] 2022-03-06T00:31:41.867 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:31:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:31:42.595 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:31:42.412407Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:31:42.412407Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:31:42.412407Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:31:42.412407Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:42.412407Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:42.412407Z", "running": 0, "size": 1}}] 2022-03-06T00:31:52.630 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:31:53.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:31:53.382 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:31:53.197815Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:31:53.197815Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:31:53.197815Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:31:53.197815Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:31:53.197815Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:31:53.197815Z", "running": 0, "size": 1}}] 2022-03-06T00:32:03.407 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:32:04.406 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:32:04.406 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:32:03.965440Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:32:03.965440Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:32:03.965440Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:32:03.965440Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:03.965440Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:03.965440Z", "running": 0, "size": 1}}] 2022-03-06T00:32:14.435 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:32:15.141 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:32:15.141 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:32:14.939522Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:32:14.939522Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:32:14.939522Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:32:14.939522Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:14.939522Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:14.939522Z", "running": 0, "size": 1}}] 2022-03-06T00:32:25.176 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:32:25.953 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:32:25.954 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:32:25.740356Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:32:25.740356Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:32:25.740356Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:32:25.740356Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:25.740356Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:25.740356Z", "running": 0, "size": 1}}] 2022-03-06T00:32:29.064 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:32:29.059076 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:32:29.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:32:29.777035 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:32:29.769+0000 7f0fd229c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:32:29.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:32:29.769+0000 7f0fd229c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:32:29.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:32:29.769+0000 7f0fd229c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:32:29.784 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:32:35.982 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:32:36.825 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:32:36.826 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:32:36.619141Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:32:36.619141Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:32:36.619141Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:32:36.619141Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:36.619141Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:36.619141Z", "running": 0, "size": 1}}] 2022-03-06T00:32:46.854 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:32:47.654 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:32:47.656 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:32:47.475545Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:32:47.475545Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:32:47.475545Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:32:47.475545Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:47.475545Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:47.475545Z", "running": 0, "size": 1}}] 2022-03-06T00:32:57.682 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:32:58.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:32:58.413 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:32:58.234486Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:32:58.234486Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:32:58.234486Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:32:58.234486Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:32:58.234486Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:32:58.234486Z", "running": 0, "size": 1}}] 2022-03-06T00:33:08.439 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:33:09.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:33:09.217 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:33:09.012732Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:33:09.012732Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:33:09.012732Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:33:09.012732Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:09.012732Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:09.012732Z", "running": 0, "size": 1}}] 2022-03-06T00:33:19.247 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:33:19.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:33:19.992 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:33:19.809443Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:33:19.809443Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:33:19.809443Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:33:19.809443Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:19.809443Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:19.809443Z", "running": 0, "size": 1}}] 2022-03-06T00:33:30.020 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:33:30.828 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:33:30.829 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:33:30.649282Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:33:30.649282Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:33:30.649282Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:33:30.649282Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:30.649282Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:30.649282Z", "running": 0, "size": 1}}] 2022-03-06T00:33:40.857 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:33:41.598 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:33:41.599 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:33:41.405792Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:33:41.405792Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:33:41.405792Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:33:41.405792Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:41.405792Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:41.405792Z", "running": 0, "size": 1}}] 2022-03-06T00:33:51.625 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:33:52.405 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:33:52.405 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:33:52.225110Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:33:52.225110Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:33:52.225110Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:33:52.225110Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:33:52.225110Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:33:52.225110Z", "running": 0, "size": 1}}] 2022-03-06T00:34:02.433 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:34:03.117 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:34:03.118 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:34:02.940910Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:34:02.940910Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:34:02.940910Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:34:02.940910Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:02.940910Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:02.940910Z", "running": 0, "size": 1}}] 2022-03-06T00:34:13.147 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:34:13.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:34:13.878 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:34:13.686527Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:34:13.686527Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:34:13.686527Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:34:13.686527Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:13.686527Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:13.686527Z", "running": 0, "size": 1}}] 2022-03-06T00:34:23.911 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:34:24.637 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:34:24.638 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:34:24.470136Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:34:24.470136Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:34:24.470136Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:34:24.470136Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:24.470136Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:24.470136Z", "running": 0, "size": 1}}] 2022-03-06T00:34:34.662 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:34:35.375 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:34:35.375 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:34:35.152222Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:34:35.152222Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:34:35.152222Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:34:35.152222Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:35.152222Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:35.152222Z", "running": 0, "size": 1}}] 2022-03-06T00:34:45.403 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:34:46.099 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:34:46.100 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:34:45.887432Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:34:45.887432Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:34:45.887432Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:34:45.887432Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:45.887432Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:45.887432Z", "running": 0, "size": 1}}] 2022-03-06T00:34:56.129 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:34:56.877 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:34:56.877 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:34:56.687352Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:34:56.687352Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:34:56.687352Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:34:56.687352Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:34:56.687352Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:34:56.687352Z", "running": 0, "size": 1}}] 2022-03-06T00:35:06.905 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:35:07.767 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:35:07.768 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:35:07.545470Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:35:07.545470Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:35:07.545470Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:35:07.545470Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:07.545470Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:07.545470Z", "running": 0, "size": 1}}] 2022-03-06T00:35:13.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:35:13.623370 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:35:14.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:35:14.311465 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:35:14.300+0000 7fbd33401380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:35:14.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:35:14.300+0000 7fbd33401380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:35:14.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:35:14.300+0000 7fbd33401380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:35:14.322 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:35:17.796 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:35:18.579 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:35:18.579 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:35:18.399931Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:35:18.399931Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:35:18.399931Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:35:18.399931Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:18.399931Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:18.399931Z", "running": 0, "size": 1}}] 2022-03-06T00:35:28.609 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:35:29.348 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:35:29.348 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:35:29.149074Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:35:29.149074Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:35:29.149074Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:35:29.149074Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:29.149074Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:29.149074Z", "running": 0, "size": 1}}] 2022-03-06T00:35:39.389 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:35:40.148 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:35:40.148 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:35:39.940268Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:35:39.940268Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:35:39.940268Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:35:39.940268Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:39.940268Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:39.940268Z", "running": 0, "size": 1}}] 2022-03-06T00:35:50.174 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:35:50.945 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:35:50.946 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:35:50.755348Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:35:50.755348Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:35:50.755348Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:35:50.755348Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:35:50.755348Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:35:50.755348Z", "running": 0, "size": 1}}] 2022-03-06T00:36:00.971 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:36:01.763 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:36:01.763 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:36:01.566526Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:36:01.566526Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:36:01.566526Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:36:01.566526Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:01.566526Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:01.566526Z", "running": 0, "size": 1}}] 2022-03-06T00:36:11.785 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:36:12.568 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:36:12.568 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:36:12.381304Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:36:12.381304Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:36:12.381304Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:36:12.381304Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:12.381304Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:12.381304Z", "running": 0, "size": 1}}] 2022-03-06T00:36:22.596 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:36:23.369 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:36:23.371 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:36:23.174998Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:36:23.174998Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:36:23.174998Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:36:23.174998Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:23.174998Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:23.174998Z", "running": 0, "size": 1}}] 2022-03-06T00:36:33.396 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:36:34.284 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:36:34.285 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:36:34.066345Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:36:34.066345Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:36:34.066345Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:36:34.066345Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:34.066345Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:34.066345Z", "running": 0, "size": 1}}] 2022-03-06T00:36:44.308 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:36:45.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:36:45.062 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:36:44.876056Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:36:44.876056Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:36:44.876056Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:36:44.876056Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:44.876056Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:44.876056Z", "running": 0, "size": 1}}] 2022-03-06T00:36:55.090 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:36:55.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:36:55.824 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:36:55.641328Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:36:55.641328Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:36:55.641328Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:36:55.641328Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:36:55.641328Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:36:55.641328Z", "running": 0, "size": 1}}] 2022-03-06T00:37:05.850 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:37:06.921 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:37:06.921 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:37:06.418683Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:37:06.418683Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:37:06.418683Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:37:06.418683Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:06.418683Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:06.418683Z", "running": 0, "size": 1}}] 2022-03-06T00:37:16.946 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:37:17.786 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:37:17.786 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:37:17.559803Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:37:17.559803Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:37:17.559803Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:37:17.559803Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:17.559803Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:17.559803Z", "running": 0, "size": 1}}] 2022-03-06T00:37:27.813 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:37:28.678 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:37:28.679 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:37:28.414076Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:37:28.414076Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:37:28.414076Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:37:28.414076Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:28.414076Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:28.414076Z", "running": 0, "size": 1}}] 2022-03-06T00:37:38.706 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:37:39.432 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:37:39.433 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:37:39.257786Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:37:39.257786Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:37:39.257786Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:37:39.257786Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:39.257786Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:39.257786Z", "running": 0, "size": 1}}] 2022-03-06T00:37:49.459 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:37:50.211 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:37:50.211 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:37:50.034676Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:37:50.034676Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:37:50.034676Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:37:50.034676Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:37:50.034676Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:37:50.034676Z", "running": 0, "size": 1}}] 2022-03-06T00:38:00.238 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:38:01.079 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:38:01.168 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:38:00.896514Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:38:00.896514Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:38:00.896514Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:38:00.896514Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:00.896514Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:00.896514Z", "running": 0, "size": 1}}] 2022-03-06T00:38:11.170 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:38:11.966 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:38:11.966 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:38:11.761197Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:38:11.761197Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:38:11.761197Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:38:11.761197Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:11.761197Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:11.761197Z", "running": 0, "size": 1}}] 2022-03-06T00:38:21.992 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:38:22.780 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:38:22.781 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:38:22.589710Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:38:22.589710Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:38:22.589710Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:38:22.589710Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:22.589710Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:22.589710Z", "running": 0, "size": 1}}] 2022-03-06T00:38:32.814 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:38:33.561 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:38:33.561 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:38:33.358612Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:38:33.358612Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:38:33.358612Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:38:33.358612Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:33.358612Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:33.358612Z", "running": 0, "size": 1}}] 2022-03-06T00:38:43.588 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:38:44.417 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:38:44.417 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:38:44.195155Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:38:44.195155Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:38:44.195155Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:38:44.195155Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:44.195155Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:44.195155Z", "running": 0, "size": 1}}] 2022-03-06T00:38:54.444 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:38:55.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:38:55.208 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:38:55.021071Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:38:55.021071Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:38:55.021071Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:38:55.021071Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:38:55.021071Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:38:55.021071Z", "running": 0, "size": 1}}] 2022-03-06T00:39:05.235 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:39:06.081 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:39:06.081 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:39:05.777874Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:39:05.777874Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:39:05.777874Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:39:05.777874Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:05.777874Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:05.777874Z", "running": 0, "size": 1}}] 2022-03-06T00:39:16.108 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:39:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:39:16.879 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:39:16.682076Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:39:16.682076Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:39:16.682076Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:39:16.682076Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:16.682076Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:16.682076Z", "running": 0, "size": 1}}] 2022-03-06T00:39:26.913 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:39:27.662 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:39:27.663 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:39:27.487110Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:39:27.487110Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:39:27.487110Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:39:27.487110Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:27.487110Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:27.487110Z", "running": 0, "size": 1}}] 2022-03-06T00:39:37.690 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:39:38.491 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:39:38.491 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:39:38.277948Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:39:38.277948Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:39:38.277948Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:39:38.277948Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:38.277948Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:38.277948Z", "running": 0, "size": 1}}] 2022-03-06T00:39:48.521 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:39:49.260 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:39:49.767 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:39:49.062898Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:39:49.062898Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:39:49.062898Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:39:49.062898Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:39:49.062898Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:39:49.062898Z", "running": 0, "size": 1}}] 2022-03-06T00:39:59.770 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:40:03.394 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:40:03.395 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:40:03.160362Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:40:03.160362Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:40:03.160362Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:40:03.160362Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:03.160362Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:03.160362Z", "running": 0, "size": 1}}] 2022-03-06T00:40:13.424 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:40:14.148 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:40:14.148 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:40:13.967149Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:40:13.967149Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:40:13.967149Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:40:13.967149Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:13.967149Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:13.967149Z", "running": 0, "size": 1}}] 2022-03-06T00:40:24.174 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:40:24.923 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:40:24.923 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:40:24.726578Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:40:24.726578Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:40:24.726578Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:40:24.726578Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:24.726578Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:24.726578Z", "running": 0, "size": 1}}] 2022-03-06T00:40:34.949 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:40:35.690 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:40:35.691 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:40:35.498956Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:40:35.498956Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:40:35.498956Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:40:35.498956Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:35.498956Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:35.498956Z", "running": 0, "size": 1}}] 2022-03-06T00:40:42.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:40:42.001483 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:40:42.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:40:42.701714 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:40:42.693+0000 7f191217b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:40:42.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:40:42.693+0000 7f191217b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:40:42.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:40:42.693+0000 7f191217b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:40:42.719 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:40:45.711 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:40:46.477 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:40:46.478 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:40:46.293269Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:40:46.293269Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:40:46.293269Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:40:46.293269Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:46.293269Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:46.293269Z", "running": 0, "size": 1}}] 2022-03-06T00:40:56.506 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:40:57.575 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:40:57.575 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:40:57.393135Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:40:57.393135Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:40:57.393135Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:40:57.393135Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:40:57.393135Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:40:57.393135Z", "running": 0, "size": 1}}] 2022-03-06T00:41:07.608 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:41:08.365 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:41:08.366 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:41:08.149521Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:41:08.149521Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:41:08.149521Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:41:08.149521Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:08.149521Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:08.149521Z", "running": 0, "size": 1}}] 2022-03-06T00:41:18.391 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:41:19.171 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:41:19.172 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:41:18.957914Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:41:18.957914Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:41:18.957914Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:41:18.957914Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:18.957914Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:18.957914Z", "running": 0, "size": 1}}] 2022-03-06T00:41:29.201 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:41:29.942 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:41:29.942 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:41:29.753714Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:41:29.753714Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:41:29.753714Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:41:29.753714Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:29.753714Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:29.753714Z", "running": 0, "size": 1}}] 2022-03-06T00:41:39.975 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:41:40.696 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:41:40.696 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:41:40.500742Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:41:40.500742Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:41:40.500742Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:41:40.500742Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:40.500742Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:40.500742Z", "running": 0, "size": 1}}] 2022-03-06T00:41:50.742 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:41:51.493 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:41:51.494 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:41:51.298940Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:41:51.298940Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:41:51.298940Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:41:51.298940Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:41:51.298940Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:41:51.298940Z", "running": 0, "size": 1}}] 2022-03-06T00:42:01.518 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:42:02.262 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:42:02.263 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:42:02.070806Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:42:02.070806Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:42:02.070806Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:42:02.070806Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:02.070806Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:02.070806Z", "running": 0, "size": 1}}] 2022-03-06T00:42:12.292 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:42:12.981 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:42:12.981 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:42:12.806133Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:42:12.806133Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:42:12.806133Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:42:12.806133Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:12.806133Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:12.806133Z", "running": 0, "size": 1}}] 2022-03-06T00:42:23.014 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:42:23.800 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:42:23.801 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:42:23.542834Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:42:23.542834Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:42:23.542834Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:42:23.542834Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:23.542834Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:23.542834Z", "running": 0, "size": 1}}] 2022-03-06T00:42:33.827 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:42:34.651 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:42:34.651 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:42:34.467503Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:42:34.467503Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:42:34.467503Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:42:34.467503Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:34.467503Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:34.467503Z", "running": 0, "size": 1}}] 2022-03-06T00:42:44.676 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:42:46.470 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:42:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:42:45.911962Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:42:45.911962Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:42:45.911962Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:42:45.911962Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:45.911962Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:45.911962Z", "running": 0, "size": 1}}] 2022-03-06T00:42:56.497 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:42:57.191 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:42:57.192 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:42:57.011947Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:42:57.011947Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:42:57.011947Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:42:57.011947Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:42:57.011947Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:42:57.011947Z", "running": 0, "size": 1}}] 2022-03-06T00:43:07.220 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:43:07.960 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:43:07.961 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:43:07.762176Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:43:07.762176Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:43:07.762176Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:43:07.762176Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:07.762176Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:07.762176Z", "running": 0, "size": 1}}] 2022-03-06T00:43:17.989 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:43:18.718 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:43:18.718 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:43:18.536374Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:43:18.536374Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:43:18.536374Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:43:18.536374Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:18.536374Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:18.536374Z", "running": 0, "size": 1}}] 2022-03-06T00:43:28.744 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:43:29.470 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:43:29.471 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:43:29.284484Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:43:29.284484Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:43:29.284484Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:43:29.284484Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:29.284484Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:29.284484Z", "running": 0, "size": 1}}] 2022-03-06T00:43:39.506 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:43:40.222 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:43:40.223 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:43:40.056170Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:43:40.056170Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:43:40.056170Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:43:40.056170Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:40.056170Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:40.056170Z", "running": 0, "size": 1}}] 2022-03-06T00:43:50.247 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:43:51.024 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:43:51.025 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:43:50.846453Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:43:50.846453Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:43:50.846453Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:43:50.846453Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:43:50.846453Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:43:50.846453Z", "running": 0, "size": 1}}] 2022-03-06T00:44:01.053 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:44:01.803 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:44:01.803 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:44:01.588895Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:44:01.588895Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:44:01.588895Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:44:01.588895Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:01.588895Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:01.588895Z", "running": 0, "size": 1}}] 2022-03-06T00:44:11.832 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:44:12.609 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:44:12.610 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:44:12.376022Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:44:12.376022Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:44:12.376022Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:44:12.376022Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:12.376022Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:12.376022Z", "running": 0, "size": 1}}] 2022-03-06T00:44:22.643 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:44:23.420 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:44:23.420 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:44:23.214042Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:44:23.214042Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:44:23.214042Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:44:23.214042Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:23.214042Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:23.214042Z", "running": 0, "size": 1}}] 2022-03-06T00:44:33.448 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:44:34.260 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:44:34.260 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:44:34.035809Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:44:34.035809Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:44:34.035809Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:44:34.035809Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:34.035809Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:34.035809Z", "running": 0, "size": 1}}] 2022-03-06T00:44:44.290 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:44:45.049 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:44:45.509 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:44:44.842203Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:44:44.842203Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:44:44.842203Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:44:44.842203Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:44.842203Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:44.842203Z", "running": 0, "size": 1}}] 2022-03-06T00:44:55.512 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:44:56.220 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:44:56.220 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:44:56.023673Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:44:56.023673Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:44:56.023673Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:44:56.023673Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:44:56.023673Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:44:56.023673Z", "running": 0, "size": 1}}] 2022-03-06T00:45:06.248 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:45:07.004 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:45:07.005 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:45:06.804894Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:45:06.804894Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:45:06.804894Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:45:06.804894Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:06.804894Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:06.804894Z", "running": 0, "size": 1}}] 2022-03-06T00:45:17.034 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:45:20.466 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:45:20.467 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:45:20.275782Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:45:20.275782Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:45:20.275782Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:45:20.275782Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:20.275782Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:20.275782Z", "running": 0, "size": 1}}] 2022-03-06T00:45:30.497 DEBUG:teuthology.orchestra.run.smithi007:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-pld5f -- ceph orch ls -f json 2022-03-06T00:45:31.333 INFO:teuthology.orchestra.run.smithi007.stdout: 2022-03-06T00:45:31.333 INFO:teuthology.orchestra.run.smithi007.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:57.000000Z", "last_refresh": "2022-03-06T00:45:31.082821Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:23:46.000000Z", "last_refresh": "2022-03-06T00:45:31.082821Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:21:13.000000Z", "last_refresh": "2022-03-06T00:45:31.082821Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "02136b44099aa21b39a2f471b260c592ebd319aa1bdc1ad273f256c38251e15f", "container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "created": "2022-03-06T00:25:25.000000Z", "last_refresh": "2022-03-06T00:45:31.082821Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-06T00:45:31.082821Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97", "last_refresh": "2022-03-06T00:45:31.082821Z", "running": 0, "size": 1}}] 2022-03-06T00:45:31.359 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:45:31.360 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:45:31.360 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-06T00:45:31.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-06T00:45:31.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-06T00:45:31.391 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-06T00:45:31.410 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:45:31.411 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:45:31.424 DEBUG:teuthology.orchestra.remote:smithi007:rook/deploy/examples/operator.yaml is 20KB 2022-03-06T00:45:31.440 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-06T00:45:31.513 INFO:teuthology.orchestra.run.smithi007.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-06T00:45:31.529 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_d0dc2fbbf3f64a2d513e2355b245d0ea9728ff97/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-06T00:45:31.529 DEBUG:teuthology.orchestra.run.smithi007:> kubectl delete -f cluster.yaml 2022-03-06T00:45:31.679 INFO:teuthology.orchestra.run.smithi007.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-06T00:45:31.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.667375 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-06T00:45:31.683 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.667512 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-06T00:45:31.688 INFO:tasks.rook.operator.smithi007.stdout:I0306 00:45:31.667903 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-06T00:45:31.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.668023 I | operator: successfully started the controller-runtime manager 2022-03-06T00:45:31.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.688981 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-06T00:45:31.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.689008 I | operator: watching all namespaces for Ceph CRs 2022-03-06T00:45:31.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.689040 I | operator: setting up schemes 2022-03-06T00:45:31.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.691818 I | operator: setting up the controller-runtime manager 2022-03-06T00:45:32.011 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.995365 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-06T00:45:32.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997672 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-06T00:45:32.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997712 I | ceph-cluster-controller: successfully started 2022-03-06T00:45:32.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997811 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-06T00:45:32.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997851 I | ceph-crashcollector-controller: successfully started 2022-03-06T00:45:32.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997888 I | ceph-block-pool-controller: successfully started 2022-03-06T00:45:32.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997915 I | ceph-object-store-user-controller: successfully started 2022-03-06T00:45:32.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997938 I | ceph-object-realm-controller: successfully started 2022-03-06T00:45:32.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997955 I | ceph-object-zonegroup-controller: successfully started 2022-03-06T00:45:32.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.997968 I | ceph-object-zone-controller: successfully started 2022-03-06T00:45:32.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998135 I | ceph-object-controller: successfully started 2022-03-06T00:45:32.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998181 I | ceph-file-controller: successfully started 2022-03-06T00:45:32.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998223 I | ceph-nfs-controller: successfully started 2022-03-06T00:45:32.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998268 I | ceph-rbd-mirror-controller: successfully started 2022-03-06T00:45:32.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998300 I | ceph-client-controller: successfully started 2022-03-06T00:45:32.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998321 I | ceph-filesystem-mirror-controller: successfully started 2022-03-06T00:45:32.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998349 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-06T00:45:32.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998369 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-06T00:45:32.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998386 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-06T00:45:32.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998405 I | ceph-bucket-topic: successfully started 2022-03-06T00:45:32.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998418 I | ceph-bucket-notification: successfully started 2022-03-06T00:45:32.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998431 I | ceph-bucket-notification: successfully started 2022-03-06T00:45:32.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.998445 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-06T00:45:32.026 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:31.999384 I | operator: starting the controller-runtime manager 2022-03-06T00:45:32.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.101121 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-06T00:45:32.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.101197 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-06T00:45:32.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.101216 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-06T00:45:32.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.106298 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-06T00:45:32.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.110464 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:32.240 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.224755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:45:32.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.239577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:45:32.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.766078 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:32.754+0000 7f20d255e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:32.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:32.754+0000 7f20d255e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:32.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:32.754+0000 7f20d255e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:32.783 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:32.808 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:32.793304 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:33.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:33.595356 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:33.582+0000 7f6f6d30a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:33.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:33.582+0000 7f6f6d30a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:33.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:33.582+0000 7f6f6d30a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:33.612 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:33.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:33.629153 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:34.296 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:34.280566 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:34.270+0000 7fef17082380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:34.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:34.270+0000 7fef17082380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:34.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:34.270+0000 7fef17082380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:34.297 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:34.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:34.343689 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:35.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:35.029807 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:35.018+0000 7f6c33e57380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:35.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:35.018+0000 7f6c33e57380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:35.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:35.018+0000 7f6c33e57380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:35.046 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:35.117 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:35.101263 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:35.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:35.803062 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:35.790+0000 7fa49a60c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:35.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:35.790+0000 7fa49a60c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:35.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:35.790+0000 7fa49a60c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:35.820 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:35.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:35.909619 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:36.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:36.558983 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:36.546+0000 7f66a3602380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:36.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:36.546+0000 7f66a3602380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:36.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:36.546+0000 7f66a3602380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:36.576 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:36.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:36.725663 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:37.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:37.400893 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:37.390+0000 7fbe4822e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:37.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:37.390+0000 7fbe4822e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:37.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:37.390+0000 7fbe4822e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:37.418 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:37.743 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:37.727503 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:38.421 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:38.405086 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:38.394+0000 7fb314335380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:38.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:38.394+0000 7fb314335380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:38.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:38.394+0000 7fb314335380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:38.422 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:39.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:39.076028 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:39.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:39.738303 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:39.726+0000 7eff8de9f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:39.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:39.726+0000 7eff8de9f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:39.756 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:39.726+0000 7eff8de9f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:39.756 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:41.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:41.027280 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:41.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:41.825473 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:41.814+0000 7fdf88f87380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:41.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:41.814+0000 7fdf88f87380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:41.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:41.814+0000 7fdf88f87380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:41.842 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:42.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:42.305596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:45:42.335 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:42.319196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:45:44.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:44.392654 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:45.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:45.135843 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:45.122+0000 7f3a293e3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:45.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:45.122+0000 7f3a293e3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:45.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:45.122+0000 7f3a293e3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:45.153 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:50.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:50.266629 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:45:50.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:50.936961 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:45:50.925+0000 7f7f114c1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:45:50.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:50.925+0000 7f7f114c1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:45:50.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:45:50.925+0000 7f7f114c1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:45:50.954 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:45:52.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:52.420617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:45:52.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:45:52.429891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:01.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:01.190769 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:46:01.866 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:01.850327 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:46:01.841+0000 7f975d36a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:46:01.867 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:46:01.841+0000 7f975d36a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:46:01.867 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:46:01.841+0000 7f975d36a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:46:01.867 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:46:02.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:02.503684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:02.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:02.518592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:12.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:12.648932 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:12.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:12.661247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:22.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:22.338434 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:46:22.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:22.785668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:22.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:22.796365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:23.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:23.148841 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:46:23.137+0000 7f92c8a1f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:46:23.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:46:23.137+0000 7f92c8a1f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:46:23.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:46:23.137+0000 7f92c8a1f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:46:23.666 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:46:32.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:32.909572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:32.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:32.922015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:43.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:43.008826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:43.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:43.020528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:53.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:53.159362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:46:53.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:46:53.171726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:03.314 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:03.298022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:03.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:03.311639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:04.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:04.115424 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:47:04.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:04.852106 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:47:04.840+0000 7fc926991380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:47:04.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:47:04.840+0000 7fc926991380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:47:04.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:47:04.840+0000 7fc926991380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:47:04.869 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:47:13.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:13.412961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:13.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:13.463228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:23.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:23.564575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:23.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:23.578829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:33.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:33.687096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:33.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:33.701497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:43.833 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:43.816463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:43.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:43.830609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:53.922 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:53.904977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:47:53.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:47:53.951754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:04.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:04.071738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:04.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:04.081214 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:14.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:14.263828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:14.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:14.273274 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:24.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:24.413632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:24.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:24.425341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:26.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:26.783574 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:48:27.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:27.449608 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:48:27.438+0000 7fb31fdde380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:48:27.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:48:27.438+0000 7fb31fdde380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:48:27.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:48:27.438+0000 7fb31fdde380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:48:27.469 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:48:34.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:34.512811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:34.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:34.527598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:44.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:44.618404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:44.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:44.632363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:54.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:54.717250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:48:54.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:48:54.731325 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:04.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:04.860203 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:04.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:04.874566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:14.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:14.962102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:14.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:14.974890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:25.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:25.088539 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:25.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:25.102445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:35.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:35.195658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:35.227 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:35.211032 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:45.292 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:45.276449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:45.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:45.290223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:55.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:55.366657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:49:55.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:49:55.380608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:05.463 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:05.446831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:05.478 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:05.461041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:15.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:15.539039 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:15.568 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:15.551906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:25.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:25.627944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:25.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:25.641688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:35.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:35.721564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:35.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:35.734742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:45.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:45.864235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:45.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:45.877889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:55.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:55.935153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:50:55.959 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:50:55.944839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:06.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:06.031376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:06.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:06.043618 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:11.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:11.304776 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:51:12.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:12.026438 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:51:12.015+0000 7ff96b959380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:51:12.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:51:12.015+0000 7ff96b959380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:51:12.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:51:12.015+0000 7ff96b959380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:51:12.041 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:51:16.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:16.151407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:16.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:16.163176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:26.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:26.296270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:26.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:26.306926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:36.431 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:36.416878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:36.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:36.430783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:46.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:46.542410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:46.568 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:46.554774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:56.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:56.637641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:51:56.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:51:56.650736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:06.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:06.774556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:06.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:06.788960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:16.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:16.890272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:16.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:16.903472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:27.065 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:27.052374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:27.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:27.065457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:37.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:37.164714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:37.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:37.179205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:47.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:47.268120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:47.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:47.311848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:57.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:57.409097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:52:57.432 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:52:57.419467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:07.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:07.524895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:07.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:07.538713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:17.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:17.639263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:17.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:17.650466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:27.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:27.714218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:27.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:27.726369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:37.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:37.812965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:37.838 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:37.825345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:47.945 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:47.932975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:47.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:47.945826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:58.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:58.148029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:53:58.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:53:58.177041 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:08.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:08.287090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:08.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:08.297442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:18.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:18.365367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:18.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:18.377971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:28.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:28.477575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:28.504 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:28.492269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:38.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:38.563575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:38.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:38.620672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:48.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:48.773788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:48.799 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:48.786270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:58.894 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:58.882214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:54:58.959 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:54:58.947637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:09.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:09.011195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:09.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:09.027703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:19.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:19.182541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:19.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:19.196320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:29.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:29.274656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:29.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:29.288901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:39.372 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:39.361703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:39.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:39.405613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:49.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:49.479852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:49.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:49.493610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:59.592 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:59.582310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:55:59.604 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:55:59.595067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:09.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:09.750813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:09.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:09.762893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:19.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:19.826955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:19.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:19.841215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:29.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:29.921194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:29.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:29.932339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:39.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:39.714280 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T00:56:40.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:40.034195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:40.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:40.049101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:40.381 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:40.371868 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T00:56:40.358+0000 7fce70950380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T00:56:40.381 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:56:40.358+0000 7fce70950380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T00:56:40.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T00:56:40.358+0000 7fce70950380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T00:56:40.382 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T00:56:50.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:50.123020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:56:50.141 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:56:50.134449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:00.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:00.212145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:00.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:00.222999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:10.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:10.323077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:10.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:10.335923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:20.450 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:20.443910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:20.462 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:20.455413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:30.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:30.541705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:30.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:30.598942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:40.851 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:40.839142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:40.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:40.854220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:51.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:51.018490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:57:51.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:57:51.034971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:01.223 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:01.216476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:01.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:01.229085 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:11.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:11.295782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:11.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:11.306516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:21.499 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:21.494550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:21.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:21.507427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:31.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:31.617054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:31.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:31.632436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:41.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:41.829228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:41.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:41.839959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:52.091 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:52.086457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:58:52.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:58:52.099442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:02.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:02.234881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:02.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:02.248197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:12.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:12.310060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:12.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:12.325488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:22.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:22.401697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:22.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:22.414190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:32.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:32.515397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:32.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:32.527822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:42.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:42.597151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:42.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:42.609273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:52.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:52.677561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T00:59:52.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 00:59:52.724911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:02.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:02.848206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:02.864 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:02.859720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:12.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:12.989357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:13.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:13.000355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:23.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:23.129935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:23.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:23.143122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:33.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:33.214465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:33.231 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:33.227096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:43.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:43.303362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:43.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:43.315893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:53.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:53.385752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:00:53.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:00:53.398917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:03.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:03.494145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:03.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:03.506263 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:13.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:13.685021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:13.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:13.700053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:23.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:23.798713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:23.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:23.812012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:33.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:33.877818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:33.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:33.927134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:44.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:44.016969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:44.034 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:44.030874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:54.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:54.104962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:01:54.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:01:54.150954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:04.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:04.258786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:04.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:04.272893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:14.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:14.372601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:14.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:14.386950 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:24.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:24.535104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:24.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:24.550121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:34.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:34.632989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:34.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:34.644517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:44.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:44.794108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:44.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:44.807734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:54.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:54.990125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:02:55.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:02:55.038815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:05.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:05.104282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:05.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:05.115918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:15.185 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:15.181259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:15.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:15.228640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:25.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:25.300506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:25.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:25.387265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:35.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:35.511150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:35.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:35.523777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:45.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:45.616644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:45.635 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:45.631737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:55.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:55.700495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:03:55.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:03:55.719529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:05.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:05.794838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:05.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:05.809505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:15.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:15.877524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:15.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:15.890172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:26.052 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:26.048730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:26.066 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:26.062426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:36.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:36.152554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:36.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:36.166434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:46.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:46.238599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:46.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:46.254191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:56.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:56.326521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:04:56.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:04:56.339996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:06.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:06.410838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:06.428 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:06.424299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:16.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:16.490945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:16.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:16.505077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:26.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:26.598968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:26.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:26.651317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:36.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:36.737646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:36.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:36.750405 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:47.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:47.011861 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:47.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:47.027198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:57.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:57.186123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:05:57.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:05:57.207377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:07.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:07.312412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:07.331 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:07.328286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:17.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:17.398008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:17.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:17.410931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:27.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:27.529718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:27.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:27.542985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:37.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:37.724080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:37.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:37.736191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:47.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:47.924508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:47.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:47.937903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:58.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:58.022944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:06:58.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:06:58.037486 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:08.155 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:08.152799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:08.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:08.167313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:18.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:18.261396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:18.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:18.276323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:28.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:28.351697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:28.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:28.362602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:35.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:35.743371 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T01:07:36.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:36.410458 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T01:07:36.398+0000 7f267950e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T01:07:36.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T01:07:36.398+0000 7f267950e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T01:07:36.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T01:07:36.398+0000 7f267950e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T01:07:36.414 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T01:07:38.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:38.427925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:38.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:38.442351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:48.589 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:48.586722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:48.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:48.598108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:58.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:58.685900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:07:58.700 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:07:58.698562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:08.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:08.806019 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:08.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:08.817784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:18.965 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:18.963081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:18.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:18.976847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:29.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:29.091068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:29.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:29.109509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:39.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:39.206584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:39.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:39.222202 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:49.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:49.305184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:49.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:49.318048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:59.434 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:59.432209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:08:59.448 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:08:59.446610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:09.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:09.550671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:09.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:09.563513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:19.677 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:19.675050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:19.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:19.688379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:29.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:29.821782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:29.834 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:29.832609 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:39.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:39.908909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:39.925 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:39.922799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:50.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:50.056652 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:09:50.070 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:09:50.069335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:00.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:00.144359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:00.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:00.156612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:10.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:10.297837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:10.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:10.311586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:20.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:20.396005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:20.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:20.408574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:30.588 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:30.586341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:30.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:30.599220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:40.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:40.714878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:40.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:40.726728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:50.857 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:50.855517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:10:50.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:10:50.869889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:01.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:01.049720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:01.064 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:01.063199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:11.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:11.168990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:11.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:11.183103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:21.258 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:21.256889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:21.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:21.271481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:31.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:31.349268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:31.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:31.363579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:41.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:41.453293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:41.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:41.464264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:51.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:51.544815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:11:51.558 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:11:51.556923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:01.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:01.649855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:01.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:01.661293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:11.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:11.764556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:11.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:11.778228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:21.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:21.906345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:21.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:21.946940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:32.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:32.090315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:32.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:32.099742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:42.211 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:42.210085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:42.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:42.223603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:52.333 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:52.332068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:12:52.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:12:52.345597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:02.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:02.468095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:02.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:02.479702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:12.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:12.749520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:12.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:12.762751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:22.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:22.872759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:22.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:22.894259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:32.969 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:32.968206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:32.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:32.981410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:43.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:43.136331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:43.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:43.148773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:53.216 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:53.214510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:13:53.231 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:13:53.229408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:03.343 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:03.341920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:03.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:03.354741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:13.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:13.463766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:13.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:13.476304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:23.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:23.546110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:23.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:23.559272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:33.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:33.630206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:33.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:33.644700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:43.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:43.706529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:43.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:43.758216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:53.868 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:53.867217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:14:53.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:14:53.881707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:03.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:03.955427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:03.971 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:03.969740 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:14.075 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:14.073955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:14.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:14.085623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:24.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:24.182523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:24.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:24.196442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:34.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:34.316009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:34.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:34.328780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:44.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:44.435211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:44.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:44.449539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:54.516 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:54.514817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:15:54.534 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:15:54.528286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:04.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:04.612563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:04.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:04.625927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:14.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:14.733009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:14.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:14.747462 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:24.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:24.892043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:24.904 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:24.902907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:34.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:34.975474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:34.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:34.989113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:45.054 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:45.052383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:45.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:45.062658 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:55.202 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:55.199464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:16:55.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:16:55.212616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:05.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:05.303803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:05.316 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:05.315874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:15.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:15.383016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:15.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:15.455283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:25.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:25.544815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:25.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:25.620295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:35.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:35.681819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:35.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:35.692895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:45.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:45.758435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:45.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:45.770281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:55.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:55.835849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:17:55.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:17:55.888680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:05.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:05.988659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:06.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:06.002573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:16.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:16.059850 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:16.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:16.072161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:26.190 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:26.188783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:26.202 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:26.201475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:36.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:36.264510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:36.278 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:36.276847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:46.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:46.339202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:46.399 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:46.398343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:56.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:56.507209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:18:56.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:18:56.520946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:06.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:06.627543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:06.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:06.639822 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:16.758 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:16.756811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:16.773 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:16.772433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:26.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:26.885408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:26.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:26.898178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:37.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:37.025759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:37.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:37.037261 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:47.163 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:47.162371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:47.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:47.173895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:57.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:57.265123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:19:57.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:19:57.280249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:07.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:07.348609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:07.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:07.360515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:17.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:17.464800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:17.480 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:17.478687 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:27.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:27.548521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:27.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:27.562534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:37.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:37.697819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:37.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:37.710266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:47.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:47.813504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:47.832 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:47.831569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:57.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:57.928569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:20:57.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:20:57.941339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:08.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:08.012279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:08.026 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:08.024829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:18.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:18.090589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:18.138 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:18.136551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:28.230 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:28.229054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:28.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:28.334929 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:38.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:38.411763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:38.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:38.424517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:48.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:48.488416 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:48.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:48.553667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:58.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:58.631430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:21:58.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:21:58.644426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:08.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:08.757334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:08.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:08.770617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:18.866 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:18.865081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:18.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:18.878877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:28.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:28.980485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:28.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:28.993060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:39.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:39.111872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:39.129 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:39.126917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:49.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:49.261240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:49.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:49.274370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:59.405 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:59.404021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:22:59.416 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:22:59.415208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:09.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:09.496225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:09.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:09.553727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:19.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:19.630598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:19.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:19.644547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:29.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:29.777121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:29.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:29.788164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:39.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:39.889271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:39.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:39.912191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:50.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:50.016961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:23:50.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:23:50.030398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:00.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:00.120599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:00.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:00.133094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:10.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:10.226381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:10.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:10.241758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:16.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:16.440223 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T01:24:17.095 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:17.093573 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T01:24:17.085+0000 7f76a1ca9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T01:24:17.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T01:24:17.085+0000 7f76a1ca9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T01:24:17.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T01:24:17.085+0000 7f76a1ca9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T01:24:17.096 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T01:24:20.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:20.340519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:20.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:20.354397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:30.473 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:30.471854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:30.484 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:30.482581 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:40.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:40.588543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:40.605 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:40.603151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:50.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:50.679184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:24:50.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:24:50.688419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:00.770 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:00.767007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:00.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:00.826741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:10.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:10.896202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:10.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:10.908724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:20.984 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:20.982066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:20.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:20.994224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:31.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:31.149976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:31.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:31.163237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:41.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:41.235822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:41.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:41.251589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:51.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:51.327394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:25:51.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:25:51.342640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:01.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:01.411586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:01.429 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:01.426244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:11.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:11.536596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:11.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:11.549122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:21.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:21.660875 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:21.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:21.675775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:31.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:31.869276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:31.886 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:31.882197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:41.969 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:41.966715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:41.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:41.980745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:52.100 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:52.097717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:26:52.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:26:52.111300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:02.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:02.173556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:02.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:02.236533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:12.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:12.302212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:12.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:12.351154 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:22.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:22.428090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:22.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:22.484596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:32.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:32.560931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:32.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:32.572695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:42.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:42.651899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:42.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:42.665440 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:52.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:52.751080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:27:52.765 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:27:52.762895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:02.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:02.855047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:02.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:02.867115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:12.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:12.984262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:13.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:12.998768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:23.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:23.095731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:23.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:23.109377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:33.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:33.206273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:33.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:33.221793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:43.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:43.319348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:43.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:43.333101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:53.415 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:53.412761 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:28:53.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:28:53.427116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:03.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:03.493664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:03.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:03.507389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:13.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:13.578908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:13.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:13.592435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:23.730 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:23.726176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:23.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:23.735608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:33.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:33.843340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:33.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:33.856673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:43.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:43.951232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:44.218 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:43.963863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:54.078 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:54.074694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:29:54.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:29:54.089856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:04.201 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:04.198250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:04.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:04.208900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:14.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:14.281230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:14.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:14.343053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:24.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:24.414627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:24.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:24.427485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:34.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:34.537440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:34.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:34.549993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:44.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:44.625368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:44.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:44.637252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:54.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:54.731721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:30:54.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:30:54.743519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:04.829 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:04.826889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:04.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:04.839155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:14.939 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:14.936409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:14.952 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:14.949560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:25.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:25.016097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:25.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:25.068661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:35.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:35.136661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:35.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:35.200837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:45.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:45.381656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:45.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:45.393825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:55.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:55.542948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:31:55.556 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:31:55.554086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:05.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:05.628165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:05.645 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:05.643148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:15.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:15.763730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:15.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:15.777112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:25.867 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:25.864902 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:25.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:25.876103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:35.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:35.985756 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:36.002 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:35.999646 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:46.094 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:46.091868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:46.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:46.106190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:56.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:56.220545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:32:56.235 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:32:56.233652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:06.300 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:06.297659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:06.314 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:06.311762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:16.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:16.420735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:16.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:16.438314 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:26.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:26.644762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:26.659 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:26.657703 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:36.729 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:36.726885 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:36.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:36.740007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:46.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:46.894167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:46.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:46.907336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:57.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:57.016374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:33:57.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:33:57.028970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:07.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:07.134618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:07.149 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:07.147556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:17.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:17.215570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:17.231 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:17.229548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:27.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:27.343291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:27.358 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:27.356640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:37.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:37.437163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:37.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:37.450567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:47.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:47.560196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:47.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:47.571807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:57.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:57.677410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:34:57.709 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:34:57.707920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:07.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:07.825137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:07.845 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:07.844144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:17.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:17.943743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:17.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:17.957781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:28.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:28.056581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:28.068 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:28.067673 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:38.174 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:38.173674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:38.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:38.223911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:48.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:48.358517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:48.371 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:48.371113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:58.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:58.436296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:35:58.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:35:58.450727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:08.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:08.529020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:08.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:08.601907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:18.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:18.693675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:18.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:18.708430 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:28.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:28.843514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:28.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:28.857812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:38.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:38.923148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:38.937 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:38.936700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:49.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:49.058530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:49.070 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:49.070546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:59.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:59.182516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:36:59.196 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:36:59.196409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:09.300 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:09.299862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:09.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:09.312176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:19.419 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:19.419064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:19.433 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:19.432882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:29.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:29.551362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:29.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:29.564954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:39.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:39.665209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:39.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:39.677693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:49.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:49.772351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:49.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:49.784884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:59.866 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:59.866561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:37:59.879 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:37:59.880256 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:10.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:10.007882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:10.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:10.021103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:20.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:20.094364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:20.106 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:20.107118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:30.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:30.206612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:30.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:30.218115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:40.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:40.319610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:40.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:40.329524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:50.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:50.391983 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:38:50.406 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:38:50.406921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:00.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:00.479604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:00.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:00.491628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:10.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:10.612792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:10.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:10.627883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:20.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:20.690058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:20.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:20.703892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:30.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:30.802331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:30.832 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:30.833293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:40.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:40.896967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:40.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:40.911171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:51.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:51.020555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:39:51.035 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:39:51.035343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:01.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:01.172454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:01.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:01.184613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:11.257 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:11.256997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:11.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:11.269960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:21.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:21.372367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:21.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:21.385890 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:31.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:31.531277 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:31.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:31.540265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:41.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:41.643280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:41.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:41.657220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:51.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:51.735446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:51.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:51.747190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:40:57.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:57.104000 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T01:40:57.792 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:40:57.790457 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T01:40:57.778+0000 7f2231983380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T01:40:57.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T01:40:57.778+0000 7f2231983380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T01:40:57.793 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T01:40:57.778+0000 7f2231983380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T01:40:57.793 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T01:41:01.864 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:01.863315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:01.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:01.877020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:12.000 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:11.998541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:12.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:12.011201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:22.135 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:22.133440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:22.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:22.148244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:32.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:32.269863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:32.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:32.284115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:42.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:42.360569 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:42.427 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:42.425098 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:52.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:52.543122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:41:52.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:41:52.552489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:02.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:02.624585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:02.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:02.639818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:12.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:12.768081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:12.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:12.778425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:22.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:22.906015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:22.922 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:22.919465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:32.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:32.983927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:33.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:33.039416 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:43.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:43.093800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:43.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:43.107487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:53.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:53.171394 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:42:53.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:42:53.185992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:03.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:03.315448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:03.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:03.326520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:13.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:13.438978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:13.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:13.451768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:23.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:23.537643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:23.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:23.551859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:33.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:33.645064 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:33.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:33.658500 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:43.748 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:43.743276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:43.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:43.756383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:53.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:53.855909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:43:53.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:43:53.891218 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:03.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:03.991373 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:04.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:04.004548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:14.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:14.083423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:14.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:14.093537 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:24.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:24.210323 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:24.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:24.223756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:34.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:34.298542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:34.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:34.339262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:44.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:44.487956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:44.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:44.498304 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:54.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:54.675590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:44:54.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:44:54.688420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:04.774 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:04.768451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:04.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:04.781333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:14.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:14.845376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:15.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:14.908278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:25.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:25.030412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:25.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:25.045259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:35.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:35.145798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:35.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:35.160193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:45.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:45.240523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:45.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:45.253134 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:55.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:55.318419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:45:55.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:45:55.335574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:05.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:05.407193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:05.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:05.419316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:15.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:15.489904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:15.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:15.503178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:25.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:25.599408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:25.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:25.610394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:35.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:35.679080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:35.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:35.692354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:45.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:45.827838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:45.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:45.840088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:55.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:55.941962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:46:55.963 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:46:55.955399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:06.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:06.033863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:06.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:06.123212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:16.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:16.268949 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:16.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:16.283192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:26.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:26.354066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:26.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:26.368245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:36.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:36.445180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:36.502 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:36.495329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:46.614 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:46.606486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:46.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:46.618720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:56.753 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:56.744962 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:47:56.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:47:56.757000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:06.874 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:06.867300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:06.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:06.880657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:17.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:17.025706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:17.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:17.037622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:27.103 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:27.095324 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:27.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:27.108760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:37.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:37.230550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:37.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:37.244239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:47.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:47.368736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:47.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:47.379847 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:57.474 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:57.466570 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:48:57.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:48:57.478395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:07.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:07.585341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:07.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:07.599828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:17.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:17.699330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:17.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:17.713873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:27.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:27.783053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:27.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:27.796374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:37.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:37.910305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:37.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:37.922483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:48.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:47.995406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:48.014 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:48.007453 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:58.133 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:58.125667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:49:58.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:49:58.137210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:08.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:08.259355 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:08.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:08.271668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:18.376 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:18.369184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:18.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:18.381211 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:28.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:28.456494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:28.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:28.504874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:38.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:38.632838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:38.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:38.646448 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:48.784 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:48.776409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:48.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:48.790418 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:58.866 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:58.858906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:50:58.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:50:58.872458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:08.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:08.941567 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:08.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:08.954760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:19.062 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:19.054404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:19.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:19.065323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:29.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:29.181422 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:29.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:29.199438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:39.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:39.332294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:39.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:39.342914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:49.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:49.502245 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:49.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:49.514276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:59.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:59.645525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:51:59.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:51:59.657110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:09.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:09.791120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:09.810 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:09.803000 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:19.882 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:19.874544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:19.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:19.884515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:29.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:29.986544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:30.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:29.998636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:40.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:40.057588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:40.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:40.069162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:50.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:50.160639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:52:50.181 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:52:50.173785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:00.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:00.280160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:00.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:00.293674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:10.399 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:10.391974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:10.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:10.405554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:20.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:20.471928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:20.492 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:20.484583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:30.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:30.603890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:30.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:30.617803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:40.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:40.677290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:40.697 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:40.689886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:50.799 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:50.791617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:53:50.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:53:50.835502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:00.956 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:00.948460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:00.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:00.959279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:11.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:11.090312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:11.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:11.102226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:21.227 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:21.219486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:21.246 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:21.239653 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:31.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:31.303368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:31.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:31.314003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:41.421 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:41.413266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:41.434 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:41.427135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:51.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:51.539228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:54:51.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:54:51.551589 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:01.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:01.640522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:01.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:01.652393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:11.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:11.831632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:11.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:11.846650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:21.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:21.935267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:21.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:21.947951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:32.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:32.051382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:32.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:32.064603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:42.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:42.168583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:42.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:42.181424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:52.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:52.252054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:55:52.274 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:55:52.266825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:02.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:02.378692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:02.398 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:02.391053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:12.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:12.460196 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:12.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:12.472378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:22.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:22.547625 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:22.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:22.562849 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:32.668 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:32.661296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:32.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:32.677796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:42.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:42.795823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:42.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:42.807839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:52.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:52.906818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:56:52.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:56:52.922339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:03.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:02.996733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:03.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:03.010995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:13.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:13.108063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:13.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:13.124009 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:23.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:23.245799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:23.268 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:23.260661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:33.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:33.379977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:33.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:33.393990 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:37.805 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:37.797712 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T01:57:38.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:38.527609 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T01:57:38.515+0000 7f122a8de380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T01:57:38.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T01:57:38.515+0000 7f122a8de380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T01:57:38.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T01:57:38.515+0000 7f122a8de380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T01:57:38.536 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T01:57:43.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:43.499851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:43.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:43.513569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:53.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:53.575309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:57:53.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:57:53.588303 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:03.666 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:03.658233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:03.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:03.724837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:13.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:13.811067 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:13.883 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:13.875664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:24.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:24.020469 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:24.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:24.034412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:34.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:34.138180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:34.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:34.154353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:44.240 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:44.233081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:44.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:44.244409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:54.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:54.311214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:58:54.384 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:58:54.376723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:04.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:04.461475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:04.516 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:04.508594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:14.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:14.659270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:14.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:14.671106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:24.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:24.743001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:24.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:24.755001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:34.834 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:34.826317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:34.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:34.840057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:44.915 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:44.907942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:44.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:44.922506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:54.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:54.983229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T01:59:55.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 01:59:54.996538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:05.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:05.120022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:05.141 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:05.133262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:15.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:15.209368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:15.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:15.221093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:25.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:25.286876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:25.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:25.299238 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:35.381 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:35.373620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:35.394 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:35.386956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:45.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:45.485170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:45.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:45.497449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:55.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:55.565655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:00:55.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:00:55.579661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:05.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:05.692914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:05.716 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:05.706893 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:15.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:15.815308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:15.836 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:15.828909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:25.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:25.894472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:25.913 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:25.905691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:36.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:36.020415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:36.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:36.033647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:46.141 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:46.133737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:46.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:46.145366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:56.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:56.239444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:01:56.257 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:01:56.249830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:06.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:06.331002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:06.353 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:06.345167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:16.434 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:16.425015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:16.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:16.437595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:26.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:26.551704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:26.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:26.568004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:36.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:36.653280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:36.704 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:36.696540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:46.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:46.809128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:46.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:46.821509 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:56.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:56.948621 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:02:56.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:02:56.959031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:07.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:07.062004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:07.085 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:07.076817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:17.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:17.148093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:17.168 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:17.159690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:27.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:27.258113 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:27.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:27.271565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:37.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:37.335771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:37.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:37.387777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:47.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:47.480575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:47.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:47.496733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:57.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:57.617403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:03:57.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:03:57.631070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:07.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:07.703214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:07.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:07.715410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:17.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:17.785718 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:17.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:17.799008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:27.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:27.871975 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:27.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:27.894410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:37.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:37.958402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:37.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:37.971598 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:48.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:48.034273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:48.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:48.047386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:58.164 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:58.155066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:04:58.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:04:58.168670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:08.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:08.277442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:08.301 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:08.292563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:18.361 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:18.351664 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:18.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:18.364999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:28.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:28.429310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:28.510 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:28.501742 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:38.607 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:38.598209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:38.621 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:38.612941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:48.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:48.701257 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:48.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:48.714434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:58.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:58.802524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:05:58.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:05:58.861428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:09.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:09.039228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:09.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:09.050392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:19.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:19.178599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:19.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:19.195142 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:29.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:29.263450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:29.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:29.280351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:39.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:39.379317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:39.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:39.393510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:49.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:49.486796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:49.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:49.500412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:59.571 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:59.561682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:06:59.586 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:06:59.577780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:09.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:09.653340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:09.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:09.665870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:19.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:19.785518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:19.808 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:19.798926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:29.933 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:29.923835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:29.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:29.935661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:40.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:40.009775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:40.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:40.019637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:50.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:50.164240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:07:50.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:07:50.177003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:00.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:00.257065 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:00.421 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:00.411597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:10.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:10.519740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:10.544 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:10.534600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:20.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:20.639856 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:20.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:20.651669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:30.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:30.759499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:30.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:30.798291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:40.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:40.909018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:40.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:40.922586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:51.039 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:51.029333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:08:51.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:08:51.041903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:01.127 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:01.117938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:01.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:01.132288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:11.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:11.204164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:11.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:11.215706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:21.337 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:21.327258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:21.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:21.336547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:31.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:31.399684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:31.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:31.414393 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:41.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:41.551546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:41.572 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:41.562629 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:51.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:51.678504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:09:51.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:09:51.693991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:01.783 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:01.772944 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:01.795 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:01.785496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:11.926 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:11.915657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:11.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:11.926111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:22.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:22.077112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:22.100 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:22.090061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:32.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:32.163433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:32.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:32.178216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:42.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:42.295296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:42.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:42.310219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:52.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:52.414807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:10:52.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:10:52.428946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:02.567 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:02.556806 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:02.582 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:02.572621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:12.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:12.648417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:12.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:12.662003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:22.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:22.730743 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:22.798 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:22.787923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:32.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:32.862752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:32.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:32.874465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:43.016 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:43.005456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:43.025 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:43.014917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:53.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:53.101046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:11:53.124 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:11:53.114117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:03.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:03.202927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:03.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:03.215026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:13.292 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:13.282014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:13.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:13.294481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:23.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:23.358398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:23.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:23.370153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:33.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:33.441432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:33.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:33.454850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:43.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:43.567690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:43.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:43.579992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:53.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:53.656392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:12:53.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:12:53.669928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:03.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:03.774674 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:03.797 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:03.787045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:13.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:13.892618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:13.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:13.906941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:24.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:23.993775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:24.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:24.058072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:34.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:34.197025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:34.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:34.208907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:44.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:44.286296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:44.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:44.298901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:54.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:54.406451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:13:54.430 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:13:54.420429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:04.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:04.508218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:04.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:04.554010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:14.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:14.652354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:14.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:14.664470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:18.547 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:18.535720 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T02:14:19.227 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:19.216820 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T02:14:19.202+0000 7f14add01380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T02:14:19.227 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T02:14:19.202+0000 7f14add01380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T02:14:19.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T02:14:19.202+0000 7f14add01380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T02:14:19.228 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T02:14:24.758 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:24.748371 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:24.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:24.761580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:34.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:34.825561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:34.849 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:34.840073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:44.999 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:44.989497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:45.067 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:45.057417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:55.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:55.274713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:14:55.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:14:55.287928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:05.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:05.400319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:05.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:05.413038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:15.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:15.488278 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:15.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:15.500384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:25.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:25.567134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:25.590 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:25.581099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:35.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:35.681081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:35.702 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:35.694002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:45.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:45.768199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:45.790 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:45.781496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:55.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:55.888934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:15:55.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:15:55.899224 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:06.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:06.038276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:06.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:06.051289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:16.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:16.149703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:16.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:16.162564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:26.275 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:26.267004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:26.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:26.281083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:36.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:36.345857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:36.371 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:36.362556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:46.491 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:46.481814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:46.502 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:46.494560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:56.579 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:56.570867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:16:56.591 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:16:56.583733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:06.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:06.703174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:06.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:06.716996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:16.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:16.794835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:16.816 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:16.808173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:26.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:26.881130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:26.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:26.895078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:36.994 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:36.986075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:37.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:37.002246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:47.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:47.106795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:47.127 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:47.119547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:57.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:57.235010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:17:57.256 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:17:57.249011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:07.364 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:07.356107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:07.375 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:07.368329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:17.487 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:17.479179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:17.499 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:17.491664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:27.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:27.592340 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:27.610 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:27.603409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:37.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:37.677951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:37.699 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:37.691967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:47.804 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:47.794160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:47.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:47.806855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:57.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:57.883726 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:18:57.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:18:57.894772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:07.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:07.962711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:07.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:07.979053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:18.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:18.102089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:18.121 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:18.113642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:28.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:28.196342 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:28.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:28.246428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:38.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:38.349271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:38.370 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:38.362787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:48.433 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:48.422766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:48.470 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:48.463196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:58.589 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:58.581854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:19:58.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:19:58.594912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:08.715 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:08.708920 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:08.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:08.720840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:18.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:18.828792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:18.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:18.837385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:28.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:28.941912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:28.959 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:28.952415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:39.066 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:39.059592 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:39.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:39.070237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:49.183 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:49.176721 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:49.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:49.190336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:59.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:59.266106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:20:59.285 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:20:59.278172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:09.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:09.362601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:09.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:09.431517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:19.526 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:19.519235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:19.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:19.535363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:29.613 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:29.606396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:29.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:29.678805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:39.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:39.847438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:39.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:39.861544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:49.929 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:49.922429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:21:49.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:21:49.936122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:00.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:00.105709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:00.124 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:00.118754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:10.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:10.182272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:10.362 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:10.355531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:20.433 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:20.427098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:20.453 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:20.447268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:30.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:30.542898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:30.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:30.555744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:40.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:40.626606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:40.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:40.684496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:50.763 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:50.757016 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:22:50.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:22:50.772826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:00.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:00.892330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:00.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:00.906737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:11.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:11.000922 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:11.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:11.064617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:21.150 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:21.141388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:21.161 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:21.155707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:31.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:31.226253 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:31.245 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:31.239241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:41.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:41.339606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:41.358 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:41.352210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:51.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:51.461227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:23:51.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:23:51.473467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:01.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:01.620409 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:01.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:01.635685 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:11.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:11.700027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:11.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:11.763282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:21.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:21.863325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:21.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:21.874532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:31.968 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:31.962484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:32.031 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:32.025025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:42.140 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:42.133912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:42.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:42.145631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:52.216 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:52.210544 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:24:52.229 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:24:52.223227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:02.293 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:02.287071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:02.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:02.299916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:12.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:12.374714 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:12.396 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:12.390210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:22.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:22.466587 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:22.486 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:22.481193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:32.568 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:32.562197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:32.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:32.575286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:42.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:42.674285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:42.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:42.689748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:52.761 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:52.755782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:25:52.776 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:25:52.771401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:02.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:02.863193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:02.880 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:02.875061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:12.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:12.945671 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:12.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:12.982508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:23.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:23.055443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:23.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:23.066555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:33.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:33.160661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:33.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:33.174516 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:43.278 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:43.273238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:43.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:43.286403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:53.389 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:53.384197 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:26:53.400 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:26:53.395707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:03.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:03.513734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:03.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:03.525711 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:13.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:13.623294 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:13.640 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:13.635019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:23.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:23.731831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:23.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:23.746298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:33.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:33.847554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:33.865 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:33.860991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:43.980 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:43.975543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:43.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:43.985605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:54.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:54.107296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:27:54.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:27:54.123331 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:04.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:04.259839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:04.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:04.275036 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:14.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:14.365045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:14.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:14.378868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:24.484 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:24.479238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:24.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:24.492721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:34.598 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:34.593909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:34.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:34.607216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:44.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:44.737795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:44.815 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:44.751119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:54.853 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:54.847815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:28:54.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:28:54.859846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:04.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:04.926977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:04.942 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:04.937860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:15.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:15.024561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:15.040 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:15.035877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:25.156 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:25.151998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:25.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:25.165007 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:35.239 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:35.235088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:35.302 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:35.297610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:45.365 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:45.360679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:45.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:45.374491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:55.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:55.478440 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:29:55.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:29:55.494288 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:05.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:05.560496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:05.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:05.573978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:15.675 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:15.670877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:15.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:15.683474 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:25.810 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:25.806120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:25.823 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:25.818952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:35.959 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:35.955520 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:35.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:35.972654 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:46.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:46.041194 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:46.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:46.055710 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:56.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:56.203006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:56.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:56.216799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:30:59.230 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:59.226425 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T02:30:59.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:30:59.881873 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T02:30:59.873+0000 7fb39a0ef380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T02:30:59.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T02:30:59.873+0000 7fb39a0ef380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T02:30:59.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T02:30:59.873+0000 7fb39a0ef380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T02:30:59.888 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T02:31:06.348 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:06.344632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:06.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:06.359820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:16.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:16.553136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:16.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:16.566220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:26.692 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:26.688680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:26.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:26.703542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:36.770 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:36.763537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:36.785 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:36.776086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:46.864 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:46.851897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:46.870 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:46.866807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:56.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:56.932366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:31:56.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:31:56.988622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:07.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:07.073360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:07.089 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:07.085532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:17.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:17.184950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:17.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:17.195316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:27.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:27.309999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:27.387 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:27.383859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:37.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:37.451369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:37.468 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:37.464733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:47.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:47.529259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:47.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:47.542215 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:57.617 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:57.613747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:32:57.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:32:57.674359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:07.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:07.744267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:07.763 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:07.759745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:17.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:17.874729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:17.889 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:17.885856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:27.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:27.992481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:28.008 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:28.005470 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:38.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:38.144643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:38.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:38.154978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:48.271 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:48.267685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:48.284 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:48.281259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:58.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:58.353816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:33:58.369 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:33:58.365748 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:08.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:08.434005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:08.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:08.447554 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:18.526 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:18.522812 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:18.593 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:18.590515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:28.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:28.723079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:28.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:28.736575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:38.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:38.809020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:38.827 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:38.824119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:48.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:48.911249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:48.988 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:48.985178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:59.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:59.114504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:34:59.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:34:59.131422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:09.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:09.197610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:09.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:09.209060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:19.309 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:19.306056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:19.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:19.319555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:29.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:29.383015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:29.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:29.437351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:39.555 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:39.551999 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:39.569 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:39.566073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:49.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:49.634417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:49.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:49.648176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:59.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:59.708145 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:35:59.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:35:59.721280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:09.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:09.823503 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:09.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:09.835762 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:19.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:19.915349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:19.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:19.927533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:30.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:30.034423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:30.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:30.048155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:40.134 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:40.131865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:40.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:40.190529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:50.314 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:50.311581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:36:50.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:36:50.325268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:00.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:00.473346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:00.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:00.486126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:10.631 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:10.627934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:10.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:10.640053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:20.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:20.714683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:20.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:20.728368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:30.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:30.832322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:30.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:30.844024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:40.987 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:40.984247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:41.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:40.998187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:51.074 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:51.070896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:37:51.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:37:51.085322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:01.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:01.186832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:01.202 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:01.199655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:11.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:11.285862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:11.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:11.301826 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:21.399 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:21.396837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:21.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:21.409668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:31.484 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:31.481048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:31.498 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:31.495352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:41.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:41.596464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:41.610 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:41.607601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:51.676 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:51.673493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:38:51.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:38:51.683324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:01.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:01.809147 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:01.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:01.821356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:11.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:11.887876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:11.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:11.900902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:22.004 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:22.001470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:22.019 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:22.015194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:32.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:32.084223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:32.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:32.134848 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:42.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:42.204331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:42.220 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:42.217721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:52.282 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:52.278987 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:39:52.295 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:39:52.293124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:02.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:02.394682 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:02.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:02.408656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:12.495 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:12.492998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:12.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:12.559948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:22.630 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:22.627709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:22.642 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:22.640184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:32.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:32.746496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:32.762 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:32.759952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:42.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:42.832541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:42.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:42.845657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:52.916 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:52.914176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:40:52.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:40:52.928957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:03.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:03.070198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:03.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:03.080697 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:13.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:13.231704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:13.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:13.246444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:23.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:23.353076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:23.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:23.365957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:33.496 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:33.493216 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:33.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:33.506691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:43.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:43.613361 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:43.636 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:43.626628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:53.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:53.725728 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:41:53.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:41:53.739109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:03.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:03.843638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:03.876 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:03.856223 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:13.965 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:13.962784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:13.979 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:13.976910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:24.099 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:24.096945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:24.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:24.107626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:34.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:34.286534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:34.304 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:34.302102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:44.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:44.401559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:44.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:44.416291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:54.522 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:54.516912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:42:54.533 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:42:54.531065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:04.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:04.646090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:04.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:04.659757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:14.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:14.729734 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:14.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:14.743831 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:24.809 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:24.807153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:24.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:24.821549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:34.936 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:34.933513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:34.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:34.946320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:45.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:45.031014 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:45.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:45.045534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:55.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:55.139558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:43:55.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:43:55.151253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:05.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:05.218456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:05.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:05.229856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:15.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:15.310330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:15.326 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:15.323719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:25.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:25.438518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:25.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:25.453175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:35.566 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:35.563889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:35.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:35.578961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:45.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:45.647876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:45.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:45.662997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:55.731 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:55.728472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:44:55.746 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:44:55.743926 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:05.821 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:05.818461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:05.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:05.871823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:15.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:15.973546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:15.989 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:15.986551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:26.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:26.056727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:26.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:26.070727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:36.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:36.143662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:36.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:36.157394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:46.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:46.267459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:46.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:46.281020 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:56.399 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:56.396513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:45:56.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:45:56.410137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:06.481 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:06.478630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:06.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:06.491790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:16.561 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:16.558432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:16.574 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:16.572137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:26.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:26.675179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:26.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:26.690680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:36.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:36.797542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:36.816 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:36.813684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:46.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:46.951011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:46.969 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:46.966840 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:57.077 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:57.075116 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:46:57.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:46:57.088234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:07.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:07.177731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:07.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:07.191006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:17.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:17.264451 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:17.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:17.277852 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:27.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:27.383745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:27.397 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:27.395608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:37.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:37.520215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:37.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:37.541068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:39.891 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:39.888777 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T02:47:40.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:40.605089 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T02:47:40.597+0000 7ff4f1651380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T02:47:40.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T02:47:40.597+0000 7ff4f1651380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T02:47:40.608 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T02:47:40.597+0000 7ff4f1651380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T02:47:40.609 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T02:47:47.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:47.644500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:47.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:47.661620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:57.778 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:57.775433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:47:57.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:47:57.787396 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:07.864 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:07.861518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:07.877 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:07.875136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:17.951 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:17.948369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:17.965 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:17.962590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:28.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:28.089989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:28.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:28.102882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:38.188 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:38.185182 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:38.201 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:38.199026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:48.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:48.304816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:48.320 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:48.317451 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:58.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:58.420402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:48:58.438 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:48:58.435130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:08.576 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:08.572737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:08.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:08.584931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:18.657 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:18.654769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:18.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:18.664424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:28.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:28.730515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:28.745 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:28.741944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:38.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:38.904613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:38.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:38.919613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:48.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:48.978528 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:48.993 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:48.990578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:59.135 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:59.131349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:49:59.148 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:49:59.145417 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:09.244 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:09.240267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:09.257 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:09.254126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:19.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:19.318789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:19.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:19.333522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:29.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:29.420081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:29.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:29.433321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:39.515 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:39.511754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:39.529 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:39.525467 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:49.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:49.596760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:49.635 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:49.631752 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:59.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:59.706739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:50:59.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:50:59.721992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:09.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:09.796493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:09.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:09.808657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:19.922 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:19.918068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:19.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:19.931310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:30.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:30.043914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:30.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:30.056534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:40.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:40.119890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:40.136 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:40.132384 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:50.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:50.299128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:51:50.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:51:50.311985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:00.391 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:00.387619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:00.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:00.399663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:10.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:10.488841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:10.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:10.503368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:20.574 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:20.570372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:20.589 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:20.584973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:30.713 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:30.709184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:30.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:30.719406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:40.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:40.821404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:40.841 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:40.837250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:50.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:50.927810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:52:50.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:52:50.939196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:01.006 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:01.002368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:01.066 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:01.061663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:11.218 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:11.214467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:11.227 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:11.223694 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:21.290 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:21.286095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:21.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:21.301081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:31.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:31.406519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:31.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:31.417772 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:41.540 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:41.535914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:41.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:41.548189 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:51.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:51.643295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:53:51.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:53:51.653783 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:01.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:01.716989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:01.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:01.730510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:11.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:11.880090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:11.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:11.895351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:22.007 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:22.001980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:22.021 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:22.016690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:32.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:32.127577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:32.144 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:32.139445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:42.242 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:42.237470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:42.257 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:42.249835 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:52.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:52.319094 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:54:52.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:54:52.334350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:02.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:02.415460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:02.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:02.472381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:12.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:12.649413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:12.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:12.661938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:22.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:22.782525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:22.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:22.797939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:32.890 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:32.884956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:32.903 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:32.898247 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:42.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:42.992412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:43.009 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:43.004321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:53.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:53.075970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:55:53.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:55:53.134668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:03.209 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:03.204606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:03.219 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:03.215195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:13.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:13.339833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:13.357 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:13.352083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:23.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:23.504129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:23.575 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:23.570631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:33.717 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:33.711677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:33.735 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:33.730262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:43.846 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:43.841128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:43.861 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:43.856290 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:53.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:53.927303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:56:53.946 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:56:53.940523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:04.016 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:04.011463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:04.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:04.023087 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:14.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:14.126107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:14.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:14.138279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:24.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:24.207316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:24.224 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:24.219107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:34.296 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:34.291276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:34.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:34.345872 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:44.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:44.418542 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:44.436 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:44.431522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:54.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:54.503068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:57:54.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:57:54.620065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:04.692 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:04.687152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:04.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:04.761903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:14.834 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:14.829214 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:14.847 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:14.841788 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:24.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:24.925081 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:24.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:24.938193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:35.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:35.043993 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:35.061 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:35.056049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:45.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:45.172662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:45.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:45.186681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:55.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:55.323202 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:58:55.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:58:55.338998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:05.445 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:05.439297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:05.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:05.452413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:15.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:15.553298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:15.571 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:15.565785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:25.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:25.669058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:25.686 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:25.681237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:35.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:35.797988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:35.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:35.808692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:45.913 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:45.907980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:45.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:45.917669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:56.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:56.012709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T02:59:56.034 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 02:59:56.028869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:06.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:06.153926 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:06.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:06.164812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:16.266 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:16.260334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:16.279 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:16.273577 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:26.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:26.384389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:26.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:26.397607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:36.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:36.518364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:36.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:36.531644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:46.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:46.593150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:46.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:46.605869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:56.702 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:56.696701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:00:56.718 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:00:56.712266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:06.832 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:06.826260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:06.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:06.836379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:16.949 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:16.943452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:16.962 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:16.957027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:27.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:27.023227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:27.041 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:27.036121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:37.155 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:37.149523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:37.169 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:37.163714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:47.268 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:47.262453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:47.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:47.276193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:57.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:57.341312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:01:57.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:01:57.358067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:07.440 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:07.434208 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:07.453 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:07.447700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:17.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:17.544787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:17.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:17.556381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:27.658 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:27.652576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:27.671 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:27.665842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:37.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:37.736693 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:37.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:37.749777 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:47.902 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:47.895905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:47.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:47.911866 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:58.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:58.009276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:02:58.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:02:58.021817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:08.153 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:08.146900 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:08.164 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:08.158058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:18.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:18.242507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:18.317 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:18.311716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:28.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:28.398001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:28.422 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:28.412123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:38.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:38.530413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:38.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:38.544633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:48.619 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:48.613415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:48.629 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:48.623443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:58.690 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:58.684537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:03:58.703 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:03:58.697650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:08.856 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:08.850480 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:08.866 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:08.860488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:18.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:18.934655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:18.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:18.952102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:20.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:20.614225 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T03:04:21.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:21.331574 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T03:04:21.319+0000 7fd340d7c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T03:04:21.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T03:04:21.319+0000 7fd340d7c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T03:04:21.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T03:04:21.319+0000 7fd340d7c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T03:04:21.339 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T03:04:29.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:29.053659 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:29.071 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:29.065644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:39.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:39.167106 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:39.186 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:39.180046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:49.260 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:49.253936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:49.273 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:49.267222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:59.378 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:59.371781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:04:59.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:04:59.384532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:09.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:09.495304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:09.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:09.512745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:19.632 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:19.626267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:19.644 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:19.638546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:29.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:29.721681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:29.742 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:29.736463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:39.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:39.837541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:39.858 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:39.852108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:49.972 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:49.965907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:05:49.986 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:05:49.980671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:00.056 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:00.050255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:00.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:00.062760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:10.140 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:10.134460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:10.154 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:10.148029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:20.262 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:20.255697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:20.274 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:20.267543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:30.373 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:30.366484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:30.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:30.380171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:40.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:40.449321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:40.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:40.496525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:50.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:50.599757 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:06:50.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:06:50.614411 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:00.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:00.682066 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:00.827 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:00.695363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:10.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:10.775455 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:10.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:10.788285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:20.866 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:20.860312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:20.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:20.871674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:30.992 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:30.985080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:31.005 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:30.998856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:41.082 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:41.075870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:41.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:41.090315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:51.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:51.160033 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:07:51.181 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:07:51.175128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:01.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:01.246481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:01.268 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:01.261691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:11.363 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:11.356507 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:11.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:11.397615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:21.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:21.459878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:21.476 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:21.470606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:31.539 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:31.533143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:31.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:31.542608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:41.684 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:41.677607 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:41.700 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:41.693532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:51.804 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:51.798075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:08:51.817 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:08:51.810635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:01.918 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:01.911555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:01.930 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:01.923851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:12.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:12.048658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:12.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:12.062540 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:22.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:22.161668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:22.182 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:22.175702 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:32.296 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:32.289148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:32.313 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:32.306277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:42.419 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:42.412521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:42.463 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:42.456536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:52.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:52.563050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:09:52.580 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:09:52.574014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:02.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:02.705562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:02.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:02.717508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:12.977 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:12.970201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:12.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:12.985229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:23.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:23.097943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:23.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:23.112146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:33.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:33.230930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:33.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:33.243919 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:43.346 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:43.339151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:43.359 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:43.352770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:53.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:53.458494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:10:53.478 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:10:53.471239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:03.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:03.542415 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:03.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:03.555472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:13.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:13.620781 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:13.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:13.658364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:23.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:23.818786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:23.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:23.831097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:33.958 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:33.950808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:33.973 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:33.966302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:44.069 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:44.062418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:44.082 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:44.076045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:54.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:54.155466 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:11:54.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:11:54.170323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:04.286 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:04.278941 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:04.300 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:04.293713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:14.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:14.380886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:14.433 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:14.426239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:24.524 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:24.516758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:24.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:24.529794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:34.641 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:34.634099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:34.655 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:34.646923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:44.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:44.752485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:44.772 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:44.765529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:54.873 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:54.866505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:12:54.896 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:12:54.889065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:04.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:04.983895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:05.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:05.084962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:15.203 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:15.196565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:15.217 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:15.210552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:25.283 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:25.274834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:25.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:25.338378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:35.428 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:35.421393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:35.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:35.435951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:45.523 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:45.516046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:45.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:45.528110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:55.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:55.599506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:13:55.620 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:13:55.613544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:05.695 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:05.687820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:05.708 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:05.702001 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:15.794 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:15.787483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:15.806 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:15.799678 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:25.872 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:25.865023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:25.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:25.904863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:35.983 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:35.975779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:35.995 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:35.988575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:46.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:46.085472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:46.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:46.099813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:56.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:56.208124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:14:56.229 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:14:56.222517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:06.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:06.320582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:06.341 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:06.333967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:16.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:16.397152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:16.418 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:16.411555 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:26.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:26.528518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:26.548 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:26.541415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:36.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:36.640835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:36.665 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:36.656164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:46.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:46.759403 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:46.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:46.772633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:56.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:56.890325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:15:56.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:15:56.902930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:07.024 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:07.016912 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:07.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:07.030172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:17.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:17.154470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:17.176 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:17.168812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:27.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:27.240576 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:27.260 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:27.253157 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:37.324 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:37.317490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:37.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:37.328669 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:47.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:47.447436 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:47.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:47.462335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:57.568 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:57.560490 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:16:57.581 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:16:57.574480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:07.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:07.641866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:07.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:07.655348 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:17.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:17.758893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:17.779 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:17.772481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:27.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:27.861397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:27.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:27.876738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:38.008 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:38.000364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:38.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:38.015309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:48.088 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:48.077843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:48.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:48.152027 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:58.249 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:58.241727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:17:58.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:17:58.252210 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:08.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:08.418378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:08.444 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:08.437273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:18.516 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:18.508545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:18.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:18.520779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:28.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:28.603888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:28.624 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:28.617695 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:38.720 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:38.712738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:38.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:38.730571 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:48.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:48.807110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:48.827 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:48.820258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:58.919 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:58.911798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:18:58.932 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:18:58.924932 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:09.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:08.994178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:09.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:09.005681 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:19.111 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:19.104687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:19.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:19.118449 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:29.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:29.221800 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:29.244 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:29.237195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:39.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:39.333813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:39.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:39.348631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:49.467 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:49.460343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:49.478 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:49.471851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:59.549 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:59.542522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:19:59.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:19:59.558820 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:09.697 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:09.690476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:09.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:09.704328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:19.808 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:19.801630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:19.822 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:19.815672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:29.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:29.947859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:29.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:29.960818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:40.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:40.031736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:40.051 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:40.045597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:50.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:50.153175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:20:50.172 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:20:50.166501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:00.259 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:00.253072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:00.270 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:00.264684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:01.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:01.376311 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T03:21:02.054 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:02.047590 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T03:21:02.036+0000 7f1c5031f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T03:21:02.054 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T03:21:02.036+0000 7f1c5031f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T03:21:02.055 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T03:21:02.036+0000 7f1c5031f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T03:21:02.055 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T03:21:10.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:10.382534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:10.401 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:10.395091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:20.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:20.483547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:20.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:20.497037 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:30.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:30.660580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:30.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:30.673454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:40.813 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:40.805878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:40.826 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:40.820029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:50.919 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:50.912936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:21:51.043 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:21:51.037682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:01.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:01.245118 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:01.263 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:01.257058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:11.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:11.329708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:11.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:11.343986 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:21.414 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:21.408150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:21.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:21.420390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:31.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:31.534698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:31.551 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:31.545250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:41.616 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:41.609558 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:41.681 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:41.674734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:51.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:51.748630 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:22:51.804 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:22:51.798216 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:01.914 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:01.903336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:01.924 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:01.918534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:11.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:11.990639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:12.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:12.007230 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:22.126 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:22.120335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:22.139 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:22.133664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:32.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:32.199271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:32.218 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:32.212298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:42.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:42.287519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:42.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:42.300708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:52.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:52.402787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:23:52.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:23:52.416863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:02.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:02.482612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:02.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:02.495773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:12.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:12.602892 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:12.623 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:12.617177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:22.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:22.685510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:22.705 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:22.698696 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:32.850 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:32.843246 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:32.862 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:32.856220 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:42.990 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:42.983676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:43.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:42.994877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:53.072 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:53.065193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:24:53.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:24:53.080879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:03.241 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:03.234589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:03.254 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:03.248362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:13.354 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:13.347738 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:13.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:13.361369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:23.441 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:23.434389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:23.455 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:23.448374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:33.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:33.521513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:33.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:33.534363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:43.626 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:43.619444 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:43.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:43.632491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:53.738 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:53.731971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:25:53.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:25:53.743584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:03.861 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:03.854088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:03.878 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:03.871250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:13.985 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:13.978367 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:13.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:13.991765 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:24.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:24.102381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:24.122 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:24.115573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:34.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:34.191097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:34.210 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:34.204002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:44.280 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:44.273047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:44.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:44.287380 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:54.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:54.401359 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:26:54.421 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:26:54.414973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:04.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:04.521013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:04.541 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:04.534456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:14.635 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:14.628563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:14.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:14.642759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:24.710 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:24.703155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:24.723 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:24.716442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:34.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:34.780959 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:34.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:34.847910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:44.917 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:44.909942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:44.927 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:44.920586 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:55.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:55.066059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:27:55.081 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:27:55.074728 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:05.171 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:05.163790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:05.251 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:05.244488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:15.368 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:15.360048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:15.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:15.372359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:25.531 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:25.524008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:25.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:25.536394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:35.646 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:35.639447 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:35.656 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:35.649808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:45.726 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:45.719354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:45.739 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:45.732354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:55.839 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:55.832198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:28:55.852 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:28:55.845647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:05.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:05.963536 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:05.982 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:05.975980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:16.101 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:16.094115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:16.114 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:16.107911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:26.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:26.208561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:26.232 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:26.225857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:36.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:36.300616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:36.322 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:36.315934 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:46.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:46.457841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:46.477 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:46.470764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:56.559 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:56.552505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:29:56.571 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:29:56.564956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:06.661 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:06.654603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:06.673 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:06.666606 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:16.788 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:16.781291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:16.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:16.794370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:26.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:26.864768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:26.884 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:26.877242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:36.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:36.932952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:36.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:36.947246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:47.023 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:47.016689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:47.038 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:47.031816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:57.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:57.098546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:30:57.121 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:30:57.114643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:07.214 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:07.206955 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:07.228 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:07.220909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:17.330 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:17.323034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:17.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:17.333179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:27.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:27.449858 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:27.471 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:27.464045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:37.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:37.566232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:37.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:37.578172 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:47.678 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:47.671321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:47.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:47.681517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:57.755 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:57.748177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:31:57.819 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:31:57.811664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:07.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:07.928170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:07.948 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:07.941984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:18.046 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:18.038696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:18.083 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:18.051347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:28.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:28.116493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:28.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:28.134406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:38.226 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:38.219215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:38.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:38.229637 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:48.307 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:48.300047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:48.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:48.310964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:58.390 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:58.382729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:32:58.403 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:32:58.396168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:08.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:08.498533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:08.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:08.513434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:18.594 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:18.586683 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:18.606 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:18.598790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:28.672 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:28.664866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:28.728 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:28.720623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:38.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:38.795561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:38.875 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:38.868617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:49.047 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:49.039332 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:49.060 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:49.053022 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:59.125 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:59.117601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:33:59.137 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:33:59.130619 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:09.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:09.201095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:09.221 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:09.213833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:19.335 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:19.327545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:19.350 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:19.343138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:29.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:29.439515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:29.461 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:29.454370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:39.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:39.556207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:39.577 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:39.570180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:49.712 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:49.704785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:49.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:49.720322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:59.801 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:59.794020 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:34:59.814 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:34:59.806980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:09.885 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:09.878382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:09.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:09.890205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:19.959 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:19.952345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:19.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:19.962935 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:30.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:30.038249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:30.086 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:30.078732 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:40.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:40.145293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:40.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:40.157964 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:50.274 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:50.266488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:35:50.288 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:35:50.281493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:00.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:00.343452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:00.410 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:00.402857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:10.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:10.472400 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:10.493 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:10.486394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:20.609 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:20.602157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:20.622 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:20.614757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:30.714 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:30.704365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:30.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:30.737277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:40.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:40.847638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:40.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:40.862479 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:50.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:50.936378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:36:50.957 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:36:50.949943 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:01.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:01.021741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:01.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:01.068264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:11.184 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:11.176914 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:11.197 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:11.190333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:21.294 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:21.287312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:21.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:21.298676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:31.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:31.416843 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:31.437 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:31.430110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:41.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:41.502372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:41.528 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:41.520764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:42.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:42.056156 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T03:37:42.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:42.743219 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T03:37:42.733+0000 7faa83e68380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T03:37:42.751 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T03:37:42.733+0000 7faa83e68380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T03:37:42.752 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T03:37:42.733+0000 7faa83e68380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T03:37:42.752 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T03:37:51.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:51.629470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:37:51.652 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:37:51.645258 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:01.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:01.742552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:01.771 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:01.764492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:11.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:11.901448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:11.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:11.916773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:22.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:22.025968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:22.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:22.041780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:32.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:32.143421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:32.164 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:32.157423 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:42.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:42.235940 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:42.255 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:42.248360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:52.379 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:52.371815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:38:52.394 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:38:52.386891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:02.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:02.480457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:02.500 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:02.493639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:12.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:12.553115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:12.573 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:12.565931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:22.679 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:22.671988 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:22.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:22.686084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:32.769 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:32.761506 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:32.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:32.775368 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:42.850 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:42.842668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:42.863 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:42.856072 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:53.048 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:53.041137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:39:53.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:39:53.052372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:03.135 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:03.125786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:03.212 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:03.204921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:13.319 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:13.311378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:13.338 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:13.330796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:23.452 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:23.445005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:23.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:23.458808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:33.598 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:33.590725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:33.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:33.603704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:43.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:43.703230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:43.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:43.717622 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:53.855 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:53.847794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:40:53.869 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:40:53.861910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:03.961 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:03.954015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:03.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:03.968371 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:14.092 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:14.084668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:14.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:14.096976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:24.234 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:24.222803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:24.243 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:24.235594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:34.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:34.307818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:34.364 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:34.356323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:44.434 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:44.426407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:44.447 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:44.438965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:54.570 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:54.562215 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:41:54.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:41:54.577668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:04.781 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:04.773684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:05.017 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:04.787165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:14.960 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:14.952984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:14.975 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:14.967729 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:25.058 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:25.051049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:25.073 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:25.064961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:35.187 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:35.179219 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:35.208 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:35.200756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:45.299 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:45.291088 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:45.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:45.303921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:55.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:55.405947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:42:55.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:42:55.417764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:05.544 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:05.537060 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:05.557 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:05.549965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:15.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:15.617884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:15.638 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:15.630836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:25.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:25.703283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:25.722 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:25.714585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:35.827 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:35.819513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:35.840 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:35.833058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:45.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:45.927224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:45.947 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:45.939909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:56.018 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:56.010312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:43:56.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:43:56.022993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:06.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:06.088402 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:06.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:06.102691 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:16.173 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:16.160831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:16.218 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:16.210191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:26.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:26.315426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:26.339 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:26.331946 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:36.426 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:36.418432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:36.439 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:36.432183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:46.518 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:46.509839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:46.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:46.522365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:56.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:56.641687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:44:56.662 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:44:56.654497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:06.733 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:06.725127 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:06.744 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:06.736985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:16.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:16.816722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:16.881 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:16.873209 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:26.944 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:26.936399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:26.996 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:26.988039 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:37.109 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:37.101822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:37.123 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:37.115505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:47.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:47.217183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:47.238 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:47.230559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:57.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:57.304642 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:45:57.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:45:57.320527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:07.412 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:07.404517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:07.423 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:07.415107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:17.538 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:17.530370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:17.553 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:17.545373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:27.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:27.617220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:27.692 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:27.684366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:37.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:37.779763 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:37.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:37.794768 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:47.898 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:47.890921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:47.909 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:47.901674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:58.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:58.013888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:46:58.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:46:58.025361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:08.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:08.143702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:08.165 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:08.158018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:18.260 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:18.250655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:18.272 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:18.265180 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:28.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:28.374327 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:28.395 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:28.387991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:38.497 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:38.489036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:38.509 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:38.501286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:48.611 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:48.603639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:48.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:48.617334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:58.741 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:58.733696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:47:58.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:47:58.743054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:08.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:08.846424 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:08.908 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:08.900237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:19.001 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:18.993789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:19.105 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:19.096672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:29.236 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:29.228330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:29.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:29.242306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:39.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:39.359136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:39.380 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:39.372234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:49.505 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:49.496921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:49.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:49.509587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:59.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:59.574948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:48:59.597 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:48:59.589656 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:09.670 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:09.662498 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:09.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:09.679213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:20.022 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:20.013839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:20.033 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:20.025735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:30.200 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:30.192028 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:30.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:30.205351 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:40.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:40.328117 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:40.347 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:40.339188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:50.408 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:50.399497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:49:50.421 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:49:50.413159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:00.495 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:00.487549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:00.508 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:00.500510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:10.583 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:10.574925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:10.595 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:10.587889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:20.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:20.678695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:20.702 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:20.694162 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:30.800 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:30.792288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:30.820 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:30.805188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:40.911 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:40.902608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:40.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:40.915596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:51.029 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:51.020688 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:50:51.042 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:50:51.034489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:01.146 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:01.138641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:01.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:01.149972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:11.227 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:11.219404 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:11.240 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:11.232361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:21.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:21.306369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:21.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:21.359662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:31.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:31.425565 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:31.446 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:31.438837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:41.516 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:41.508336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:41.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:41.522289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:51.600 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:51.591938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:51:51.732 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:51:51.604012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:01.685 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:01.677053 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:01.698 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:01.690364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:11.775 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:11.767074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:11.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:11.781473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:21.895 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:21.886577 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:21.906 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:21.898616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:32.015 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:32.006751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:32.028 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:32.020676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:42.131 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:42.122685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:42.147 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:42.139201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:52.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:52.207166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:52:52.230 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:52:52.221845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:02.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:02.301821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:02.323 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:02.314941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:12.420 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:12.411817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:12.454 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:12.443563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:22.564 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:22.555845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:22.578 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:22.570064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:32.693 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:32.684823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:32.706 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:32.697722 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:42.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:42.772315 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:42.835 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:42.827201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:52.943 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:52.934798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:53:52.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:53:52.947517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:03.032 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:03.024031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:03.096 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:03.088146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:13.166 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:13.157824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:13.180 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:13.171854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:22.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:22.750711 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T03:54:23.267 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:23.259082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:23.281 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:23.272834 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:23.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:23.480103 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T03:54:23.466+0000 7f9d655a7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T03:54:23.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T03:54:23.466+0000 7f9d655a7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T03:54:23.490 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T03:54:23.466+0000 7f9d655a7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T03:54:23.490 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T03:54:33.388 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:33.379996 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:33.404 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:33.395922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:43.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:43.503493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:43.525 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:43.517018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:53.637 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:53.628947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:54:53.650 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:54:53.641723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:03.727 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:03.718848 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:03.740 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:03.731560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:13.812 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:13.803150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:13.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:13.816374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:23.931 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:23.923015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:23.945 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:23.937236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:34.008 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:33.999275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:34.020 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:34.012143 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:44.142 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:44.133100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:44.152 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:44.143911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:54.235 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:54.226248 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:55:54.247 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:55:54.239336 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:04.546 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:04.533076 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:04.552 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:04.544276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:14.768 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:14.760736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:14.782 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:14.774333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:24.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:24.946280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:24.970 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:24.962817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:35.087 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:35.079497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:35.098 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:35.091096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:45.179 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:45.172264 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:45.193 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:45.185878 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:55.306 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:55.298931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:56:55.318 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:56:55.311117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:05.428 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:05.421184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:05.442 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:05.436204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:15.519 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:15.512961 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:15.530 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:15.524086 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:25.628 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:25.621679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:25.639 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:25.633549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:35.736 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:35.729759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:35.750 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:35.744733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:45.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:45.824901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:45.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:45.837602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:55.938 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:55.932532 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:57:55.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:57:55.947046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:06.080 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:06.074881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:06.093 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:06.088208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:16.178 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:16.173103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:16.192 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:16.187457 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:26.345 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:26.340572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:26.355 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:26.350282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:36.516 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:36.511891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:36.545 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:36.540515 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:46.612 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:46.607956 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:46.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:46.621299 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:56.711 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:56.707268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:58:56.725 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:58:56.720844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:06.920 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:06.916213 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:06.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:06.931092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:17.097 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:17.093486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:17.110 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:17.107050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:27.207 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:27.203430 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:27.225 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:27.221721 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:37.315 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:37.311826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:37.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:37.324320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:47.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:47.399036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:47.417 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:47.413888 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:57.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:57.528645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T03:59:57.543 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 03:59:57.540383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:07.654 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:07.651269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:07.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:07.664838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:17.764 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:17.761299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:17.777 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:17.775044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:27.844 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:27.839874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:27.854 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:27.851575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:37.954 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:37.951417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:37.966 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:37.963525 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:48.036 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:48.033717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:48.050 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:48.047936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:58.115 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:58.112744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:00:58.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:00:58.167889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:08.291 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:08.289267 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:08.303 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:08.301763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:18.370 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:18.368582 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:18.383 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:18.381897 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:28.457 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:28.455857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:28.469 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:28.467810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:38.587 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:38.585849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:38.602 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:38.601115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:48.674 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:48.672531 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:48.687 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:48.686109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:58.766 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:58.765228 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:01:58.780 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:01:58.779494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:08.857 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:08.856251 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:08.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:08.870293 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:19.045 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:19.043884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:19.057 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:19.056562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:29.194 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:29.193564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:29.206 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:29.205797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:39.312 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:39.311826 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:39.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:39.325090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:49.456 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:49.455325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:49.466 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:49.466265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:59.585 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:59.584700 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:02:59.598 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:02:59.597854 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:09.680 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:09.675146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:09.689 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:09.689289 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:19.807 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:19.806836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:19.897 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:19.820942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:29.892 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:29.892546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:29.907 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:29.907841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:40.003 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:40.003413 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:40.013 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:40.014111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:50.112 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:50.112550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:03:50.120 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:03:50.121270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:00.204 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:00.205162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:00.215 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:00.216226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:10.329 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:10.330142 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:10.342 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:10.343108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:20.465 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:20.466370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:20.479 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:20.480061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:30.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:30.551122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:30.562 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:30.563561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:40.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:40.664150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:40.673 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:40.674676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:50.795 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:50.796022 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:04:50.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:04:50.826188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:00.888 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:00.889362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:00.899 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:00.901382 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:10.997 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:10.998262 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:11.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:11.014096 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:21.143 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:21.144822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:21.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:21.153460 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:31.222 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:31.218380 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:31.278 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:31.279940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:41.356 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:41.358218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:41.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:41.415040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:51.517 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:51.518464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:05:51.532 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:05:51.533802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:01.633 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:01.635112 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:01.648 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:01.650280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:11.767 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:11.769059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:11.781 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:11.783994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:21.928 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:21.930563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:21.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:21.943280 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:32.064 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:32.066396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:32.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:32.079200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:42.159 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:42.161739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:42.231 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:42.233875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:52.351 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:52.354365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:06:52.366 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:06:52.368836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:02.520 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:02.523362 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:02.560 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:02.563309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:12.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:12.654453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:12.664 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:12.667330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:22.773 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:22.776612 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:22.787 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:22.790781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:32.893 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:32.896681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:32.910 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:32.913903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:43.027 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:43.030545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:43.059 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:43.062429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:53.163 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:53.167086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:07:53.175 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:07:53.179097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:03.248 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:03.251463 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:03.305 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:03.308572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:13.411 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:13.415291 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:13.425 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:13.428916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:23.501 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:23.504818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:23.512 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:23.516513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:33.615 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:33.618581 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:33.627 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:33.631278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:43.707 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:43.711290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:43.721 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:43.724995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:53.824 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:53.828258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:08:53.837 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:08:53.841273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:03.976 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:03.980590 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:03.991 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:03.995235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:14.104 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:14.108011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:14.116 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:14.120652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:24.189 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:24.192896 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:24.201 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:24.205358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:34.336 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:34.340626 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:34.367 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:34.371604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:44.507 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:44.511905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:44.516 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:44.521403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:54.625 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:54.629672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:09:54.653 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:09:54.658178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:04.724 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:04.729236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:04.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:04.742155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:14.848 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:14.853150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:14.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:14.865123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:25.012 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:25.016918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:25.035 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:25.035686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:35.119 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:35.123733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:35.132 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:35.137514 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:45.250 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:45.255109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:45.265 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:45.270724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:55.374 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:55.379464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:10:55.386 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:10:55.391232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:03.482 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:03.487032 I | op-mon: parsing mon endpoints: a=10.102.190.159:6789,b=10.98.23.111:6789,c=10.109.72.196:6789 2022-03-06T04:11:04.157 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:04.162200 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-06T04:11:04.152+0000 7ff2cc9d5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-06T04:11:04.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T04:11:04.152+0000 7ff2cc9d5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-06T04:11:04.158 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06T04:11:04.152+0000 7ff2cc9d5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-06T04:11:04.158 INFO:tasks.rook.operator.smithi007.stdout:couldn't init storage provider. : exit status 5 2022-03-06T04:11:05.489 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:05.494459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:05.503 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:05.508617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:15.572 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:15.577166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:15.584 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:15.589058 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:25.647 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:25.652377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:25.663 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:25.668596 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:35.789 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:35.794370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:35.802 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:35.807633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:45.998 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:46.003647 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:46.010 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:46.015930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:56.107 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:56.112553 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:11:56.118 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:11:56.124469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:06.198 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:06.204388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:06.252 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:06.257976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:16.344 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:16.350420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:16.358 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:16.364294 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:26.435 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:26.441244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:26.448 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:26.454406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:36.535 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:36.541705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:36.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:36.555876 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:46.635 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:46.641274 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:46.649 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:46.655677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:56.760 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:56.765937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:12:56.803 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:12:56.809635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:06.941 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:06.947075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:06.955 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:06.961761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:17.063 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:17.069526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:17.076 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:17.083038 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:27.162 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:27.168584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:27.177 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:27.183604 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:37.287 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:37.294295 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:37.297 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:37.304056 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:47.409 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:47.415452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:47.424 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:47.429828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:57.563 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:57.570111 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:13:57.577 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:13:57.584296 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:07.651 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:07.657977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:07.667 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:07.669480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:17.737 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:17.744030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:17.749 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:17.755797 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:27.843 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:27.850173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:27.860 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:27.862137 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:37.923 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:37.929712 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:37.940 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:37.943185 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:48.030 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:48.037685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:48.044 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:48.051332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:58.108 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:58.115398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:14:58.488 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:14:58.129774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:08.199 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:08.206771 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:08.213 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:08.220626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:18.311 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:18.318477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:18.325 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:18.333081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:28.402 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:28.409171 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:28.413 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:28.421241 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:38.521 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:38.528730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:38.536 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:38.543699 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:48.601 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:48.609166 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:48.615 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:48.622719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:58.734 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:58.742232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:15:58.759 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:15:58.761628 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:08.830 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:08.837821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:08.842 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:08.850665 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:18.953 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:18.960523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:18.967 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:18.974921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:29.090 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:29.097805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:29.102 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:29.110401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:39.170 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:39.178434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:39.191 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:39.194226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:49.276 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:49.284374 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:49.289 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:49.297236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:59.494 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:59.502502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:16:59.506 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:16:59.514744 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:09.660 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:09.668283 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:09.747 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:09.754832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:19.871 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:19.879357 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:19.887 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:19.895502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:30.037 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:30.044829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:30.049 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:30.057973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:40.310 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:40.318388 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:40.321 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:40.329886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:50.382 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:50.390438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:17:50.393 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:17:50.401942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:00.459 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:00.468085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:00.472 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:00.481070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:10.554 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:10.562476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:10.565 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:10.574310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:20.688 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:20.696369 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:20.701 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:20.709592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:30.811 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:30.819789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:30.825 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:30.833520 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:40.922 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:40.930596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:40.935 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:40.943956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:51.011 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:51.019497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:18:51.026 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:18:51.035166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:01.128 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:01.136459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:01.151 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:01.159766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:11.253 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:11.261825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:11.264 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:11.273567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:21.328 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:21.336935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:21.340 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:21.349236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:31.451 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:31.460300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:31.464 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:31.473485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:41.537 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:41.545852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T04:19:41.550 INFO:tasks.rook.operator.smithi007.stdout:2022-03-06 04:19:41.559010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-06T06:39:20.452 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-06T06:39:20.480 DEBUG:teuthology.task.console_log:Killing console logger for smithi007 2022-03-06T06:39:20.481 DEBUG:teuthology.task.console_log:Killing console logger for smithi017 2022-03-06T06:39:20.482 DEBUG:teuthology.task.console_log:Killing console logger for smithi066 2022-03-06T06:39:20.483 DEBUG:teuthology.exit:Finished running handlers